Video analysis method and device

Abstract

The invention provides a video analysis method and device. The method comprises the following steps: determining a fusion video to be analyzed; wherein the fused video comprises a live video with a three-dimensional rendering effect; acquiring a first index and a second index corresponding to the fused video to be analyzed; wherein the first index is used for representing a stability index of the live video, and the second index is used for representing a stability index of three-dimensional rendering; different index analysis methods are utilized to determine the lag reason of the fusion video to be analyzed; and customizing corresponding stability measures according to the lagging reason.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a video analysis method and device. Background Art

[0002] With the development of live video technology, live video technology has been widely used in more and more scenarios.

[0003] In some live video broadcast scenarios, to enhance the viewing experience and meet audience needs, live video is blended with 3D rendered video for playback. During playback, various factors can inevitably cause video playback to freeze. The cause of freezes in the fused video can be difficult to pinpoint. It could be due to issues with the live video or the 3D rendered video. Troubleshooting freezes in the fused video is complex, and it can sometimes be difficult to pinpoint the cause. Summary of the Invention

[0004] The present disclosure provides a video analysis method and device.

[0005] According to a first aspect of the present disclosure, a video analysis method is provided. The method specifically comprises: determining a fused video to be analyzed; wherein the fused video includes a live video with a three-dimensional rendering effect; obtaining a first indicator and a second indicator corresponding to the fused video; wherein the first indicator is used to represent the stability index of the live video, and the second indicator is used to represent the stability index of the three-dimensional rendering; analyzing the first indicator and / or the second indicator to determine the cause of freezes in the fused video to be analyzed; and customizing corresponding stability measures based on the freezes.

[0006] Based on the above, we can see that by analyzing the live video and 3D rendering in the fused video, we can determine a first indicator for the stability of the live video and a second indicator for the stability of the 3D rendering. We then use these first and second indicators to conduct targeted stability analyses of the live video and 3D rendering, respectively, to identify the cause of video instability. With clear analysis indicators, stability analysis of the fused video is more accurate and efficient.

[0007] According to at least one embodiment of the present disclosure, obtaining a first indicator and a second indicator in a fused video to be analyzed includes: obtaining a first indicator for characterizing the stability of a live video from a video player; and obtaining a second indicator for characterizing the stability of a three-dimensional rendering from a three-dimensional rendering engine.

[0008] According to at least one embodiment of the present disclosure, a first indicator for characterizing the stability of a live video is obtained from a video player, including: obtaining a playback freeze event and an event timestamp corresponding to the freeze event from the video player through a specified interface or network layer; and obtaining the first indicator based on the live broadcast freeze event and the event timestamp.

[0009] According to at least one embodiment of the present disclosure, different indicator analysis methods are used to determine the cause of freezes in the fused video to be analyzed, including: determining the cache time and playback time of the live video based on an event timestamp; calculating the difference between the cache time and the playback time as the delay time; counting the number of freezes during the live video playback process with a delay time less than zero; if the number of freezes is greater than or equal to a first count threshold, determining that the live video has freezes.

[0010] According to at least one embodiment of the present disclosure, different indicator analysis methods are used to determine the cause of freezes in the fused video to be analyzed, including: obtaining freeze events in the current live broadcast; counting the number of low speeds in which the network speed parameter is lower than the network speed threshold during the freeze event; if the number of low speeds is greater than or equal to a second number threshold, determining that freezes have occurred in the live video in the fused video.

[0011] According to at least one embodiment of the present disclosure, a second indicator for characterizing the stability of three-dimensional rendering is obtained from a three-dimensional rendering engine, including: in the three-dimensional rendering engine, driving a rendering loop through recursive calls; using a callback function to record a timestamp based on the rendering result; and using the difference between the timestamps of two consecutive frames to calculate a second indicator representing the frame rate.

[0012] According to at least one embodiment of the present disclosure, different indicator analysis methods are used to determine the cause of freezes in the fused video to be analyzed, including: obtaining freeze events in the current live broadcast; counting the number of low FPS times in which the FPS is lower than an FPS threshold during the freeze events; if the number of low FPS times is greater than or equal to a third number threshold, determining that freezes have occurred in the three-dimensional rendering of the fused video.

[0013] According to at least one embodiment of the present disclosure, obtaining a first indicator and a second indicator in a fused video to be analyzed includes: in response to a playback request of the fused video, obtaining a first indicator reported by a video player in the form of a network performance array when the fused video is played; obtaining a second indicator reported by a three-dimensional rendering engine in the form of a rendering performance array; clearing the network performance array and the rendering performance array after successful reporting; and terminating reporting after the live broadcast duration of the fused video exceeds a time threshold.

[0014] According to at least one embodiment of the present disclosure, corresponding stability measures are customized according to the cause of the lag, including: if the cause of the lag is determined to be live broadcast lag, the customized stabilization measures include: optimizing network configuration, increasing bandwidth and adjusting video bit rate at least one; if the cause of the lag is determined to be 3D rendering lag, the customized stabilization measures include: reducing model complexity and improving GPU performance at least one.

[0015] According to a second aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the method described in the first aspect of any embodiment of the present disclosure.

[0016] According to a third aspect of the present disclosure, a readable storage medium is provided, in which execution instructions are stored. When the execution instructions are executed by a processor, they are used to implement the method described in the first aspect of any embodiment of the present disclosure.

[0017] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the method according to the first aspect of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0019] Figure 1 A schematic diagram of a video analysis method provided by the present invention.

[0020] Figure 2 A schematic diagram illustrating the video analysis process of the present disclosure.

[0021] Figure 3 The present invention is a schematic block diagram of a video analysis device according to an embodiment of the present invention.

[0022] Figure 4 The present invention is a block diagram showing the structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present disclosure is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are intended only to illustrate the relevant content and are not intended to limit the present disclosure. It should also be noted that, for ease of description, only the portions relevant to the present disclosure are shown in the accompanying drawings.

[0024] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] As live video technology becomes more widespread, it's finding applications in an increasing number of scenarios. In some specialized scenarios, to enhance the viewing experience, live video is rendered in 3D, creating a fused video with a 3D scene. This fused video not only meets the needs of live viewers, but also provides a superior 3D viewing experience.

[0026] However, during fusion video playback, various uncertainties can easily cause lags. Analyzing the causes of fusion video lags is complex, as a single indicator alone cannot pinpoint the specific component causing the lag. Further, specific analysis based on the actual situation is required, making the entire analysis process cumbersome. Therefore, a solution is urgently needed to accurately and quickly analyze the causes of fusion video lags.

[0027] For the convenience of description and to make the technical solutions of the specific embodiments of the present disclosure easier to understand, before describing the image synthesis method implemented in the present disclosure, the technical terms involved in the specific embodiments of the present disclosure are explained as follows.

[0028] Live video: refers to video streams captured, encoded, transmitted, and played in real time by video capture devices such as cameras. Live video playback requires high network performance.

[0029] 3D rendered video: This is a 3D video generated by real-time rendering of specified content using a 3D rendering model within a 3D rendering engine. The rendering quality is directly related to the 3D rendering model, while the rendering speed is affected by factors such as the complexity of the 3D rendering model and GPU performance.

[0030] Fusion video: refers to the use of a 3D rendering engine to render live video in real time to generate a real-time dynamic picture.

[0031] To facilitate understanding of the fused video generation process, the following example uses a situation where multiple cameras in a room simultaneously capture live video and fuse it after 3D rendering on the client.

[0032] Multi-camera calibration and spatial alignment. Calibration purpose: Determine the intrinsic parameters (focal length, distortion coefficient) and extrinsic parameters (position, rotation angle) of each camera to establish a unified spatial coordinate system.

[0033] Calibration method: Use a calibration board (such as a checkerboard) to capture multi-view images and calculate intrinsic parameters and distortion parameters using tools such as OpenCV. By projecting multiple views onto known objects (such as fixed markers in a room), the relative positions and rotation matrices between the cameras are calculated to achieve spatial alignment.

[0034] Next, multi-view 3D reconstruction is performed, including static and dynamic scene reconstruction. For static scene reconstruction, structured light or stereo matching algorithms (such as Semi-Global Matching) are used to extract depth information from multi-view images and generate a dense point cloud or mesh model. Multi-view stereo (MVS) technology is used to fuse multi-view data to improve reconstruction accuracy. For dynamic scene reconstruction, real-time dynamic 3D reconstruction techniques (such as deep learning-based NeRF or VolumeDeform) are used to estimate dynamic deformation and update the model using multi-frame video streams.

[0035] Next, the video streams captured by the cameras are synchronized and dynamically fused, including time synchronization and texture fusion. Time synchronization involves aligning the video streams from multiple cameras through hardware synchronization (such as Genlock) or software timestamps to ensure frame-level synchronization. Texture fusion involves mapping the textures of the multi-view videos onto the surface of the 3D rendered model, eliminating seams and ghosting through weighted fusion (e.g., assigning weights based on view distance) or illumination consistency correction.

[0036] The fused video is rendered in real time. For example, game engines (such as Unity / UnrealEngine) or WebGL can be used to combine 3D rendered models with dynamic textures, supporting real-time shadows, light reflections, and other effects. For real-time scenarios (such as live broadcasts), the latest video frame is updated to the texture map corresponding to the camera's perspective through asynchronous streaming to ensure low latency.

[0037] After the above videos are captured and fused in real time, a 3D-like fused video is presented to the user on their mobile phone. Users can watch the fused video on their mobile phone and also see details of the room in 3D, creating an immersive viewing experience.

[0038] Figure 1 The following is a flow chart of a video analysis method provided by the present disclosure. Figure 1 The method shown includes steps 101 to 104. The method can be executed by an electronic device such as a server (a local server or a cloud server).

[0039] Specifically, Figure 1 The method shown includes: Step 101: determining a fused video to be analyzed; wherein the fused video includes a live video with a three-dimensional rendering effect.

[0040] In practical applications, to achieve a more comprehensive video capture effect, multiple video capture devices (such as cameras or mobile phones with video capture capabilities) can be used to capture the same content in real time to generate live video. The captured live video is then imported into a 3D rendering engine, where it is rendered using a 3D rendering model to create a fused video.

[0041] For example, in a room undergoing renovation, multiple cameras are deployed to capture real-time video. The homeowner who wants to view the renovation progress and details in real time can do so through live video. However, live video lacks 3D effects, and the homeowner (the viewer of the live video) cannot see the desired room content. Therefore, the live video is sent to the user's mobile phone, where it is rendered in 3D and played directly on the phone. It should be noted that the user's mobile phone (the terminal device that plays the fused video) is equipped with a 3D rendering engine and 3D rendering model, which enables real-time rendering of the received live video.

[0042] It should be noted that the 3D rendering engine can be deployed locally on the client, that is, after the live video is sent to the 3D rendering engine, it is rendered in real time locally on the client, and the rendered fusion video is generated locally and can be played directly for users to watch.

[0043] Step 102: Obtain a first indicator and a second indicator corresponding to the fused video; wherein the first indicator is used to characterize the stability index of the live video, and the second indicator is used to characterize the stability index of the three-dimensional rendering.

[0044] During the fusion video playback process, the server will analyze the playback stability of the fusion video. During the analysis, in order to accurately identify the cause of the fusion video freeze, it is necessary to collect the first indicator and the second indicator related to the fusion video.

[0045] The first metric mentioned here can be understood as an indicator used to characterize the stability of live video, which is usually a network-related parameter. It should be noted that the smoothness of live video is mainly affected by network conditions, but this does not directly change the FPS of the video itself. The client only needs to decode and play the video, and the smoothness depends more on the network transmission quality rather than the local rendering capabilities.

[0046] The second metric mentioned here can be understood as an indicator used to characterize the stability of the 3D rendering process. This metric is typically frames per second (FPS), which is the number of times the image is updated per second. It's important to note that 3D rendering is computed in real time, requiring hardware to process each frame. A low FPS can cause visual lag, and the user experience in the virtual scene (such as rotating the viewport or moving objects) becomes unsmooth. Limited by local computing power, such as scene complexity, model detail, or hardware performance, FPS directly reflects the efficiency of the rendering engine and is therefore a core metric for evaluating 3D rendering smoothness. The FPS of live video is typically determined by the video source (e.g., a camera), set during encoding (e.g., 30 FPS or 60 FPS), and remains relatively fixed during transmission and playback.

[0047] Unlike 3D rendering of static images, using a 3D rendering model to render and fuse live video requires high real-time performance, which in turn places high demands on network performance. During live video transmission, the camera first transmits the raw live video captured to the 3D rendering engine for rendering. This process is easily affected by network speed.

[0048] After receiving the live video, the 3D rendering engine uses the 3D rendering model to render the live video, thereby obtaining a fused video with a 3D effect. This means that the stability of the 3D video rendering process needs to be analyzed.

[0049] Step 103: Analyze the first indicator and / or the second indicator to determine the cause of the freeze in the fused video to be analyzed.

[0050] After obtaining the first indicator and the second indicator respectively through the methods described above, the corresponding indicator analysis methods are used to perform targeted analysis.

[0051] The first indicator mentioned here can be a network-related performance indicator, such as network speed and network bandwidth. During the analysis, targeted analysis is performed on each network performance indicator, analyzing whether the network speed meets the standard and whether the network bandwidth meets the standard.

[0052] The second indicator here refers to the FPS during 3D rendering. During analysis, you can determine whether the FPS of the 3D rendering collected in real time meets the standard.

[0053] When analyzing the cause of the lag, since the prerequisite for the 3D rendering process is to smoothly receive the live video, if the live video is stuck due to network problems, it will also affect the efficiency of the 3D rendering.

[0054] Therefore, when analyzing the cause of the freeze of the fused video, it is necessary to comprehensively analyze the analysis results of the first indicator and the second indicator to finally determine the cause of the freeze of the fused video.

[0055] Step 104: Customize corresponding stability measures based on the cause of the lag.

[0056] As mentioned above, after determining the cause of fused video freezes through the above solution, we can further implement appropriate stability measures to address the issue. For example, if live video freezes, we can improve network performance parameters. If 3D rendering freezes, we can improve the 3D rendering model or GPU.

[0057] In one or more embodiments of the present disclosure, obtaining the first indicator and the second indicator in the fused video to be analyzed includes: obtaining the first indicator for characterizing the stability of the live video from the video player. Specifically, the first indicator includes: obtaining a playback freeze event and an event timestamp corresponding to the freeze event from the video player through a specified interface or network layer; obtaining the first indicator based on the live freeze event and the event timestamp. Obtaining the second indicator for characterizing the stability of the three-dimensional rendering from the three-dimensional rendering engine. Specifically, the second indicator includes: In practical applications, the first metric is obtained from the video player, for example, through the video player's Video API or network layer. Specifically, key events can be used to monitor the network data download status and whether the player's buffer is exhausted to determine network download speed and whether there is any lag. The player's built-in statistics can also be used to calculate the download bitrate (and thus download speed). Alternatively, the time interval (Δt) between two consecutive download events (progress) and the amount of buffered data (ΔData) can be recorded to calculate the download speed (and thus the network speed).

[0058] To obtain network-related performance metrics at the network layer, you can use the browser's NetworkInformation API. Alternatively, you can analyze packet capture, for example, using Wireshark to analyze the throughput per unit time.

[0059] The second metric is the FPS obtained from the 3D rendering engine. In practice, there are various ways to obtain this information. For example, you can use the engine's built-in performance monitoring tools. Open the Unity Editor, select the Rendering tab in the Profiler window, and you'll see the real-time FPS value. You can also use third-party performance monitoring tools, such as MSI Afterburner.

[0060] The above solution obtains the second indicator from the 3D rendering engine and the first indicator from the video player, effectively distinguishing the different sources of the indicators and thus more accurately analyzing the cause of the lag.

[0061] In one or more embodiments of the present disclosure, the first indicator and / or the second indicator are analyzed to determine the cause of the freeze in the fused video to be analyzed, including: obtaining freeze events in the current live broadcast, and event timestamps corresponding to the freeze events; determining the cache time and playback time of the live video based on the event timestamps; calculating the difference between the cache time and the playback time as the delay time; counting the number of freezes with a delay time less than zero during the playback of the live video; if the number of freezes is greater than or equal to a first count threshold, determining that the live video in the live video is freezed.

[0062] The event timestamp referred to here refers to the timestamp corresponding to the time when the freeze event occurred. This includes the live video cache time (i.e., the time when the live video was stored in the cache) and the live video playback time (i.e., the time when the live video was retrieved from the cache). Generally speaking, if the network is normal, the cache has sufficient data, and the latency is greater than zero, the live video will not freeze. If the network freezes, it means that there is insufficient data in the cache (i.e., the latency is less than or equal to zero), and the playback progress has exceeded the cache progress. This is because the network freeze has caused an insufficient supply of live video.

[0063] In actual applications, network lag may occur due to accidental factors. This is an uncontrollable and uncertain situation. Therefore, to accurately identify the cause of lag, it is necessary to analyze the problem after multiple lags occur. In other words, the number of lags needs to be counted. When the number of lags is greater than or equal to the first count threshold, it is considered a lag event in the live video.

[0064] Based on the above public solution, after using the event timestamp to detect the freeze event, the live video cache time and playback time can be further used to determine whether the live video freeze has occurred. The above solution can accurately identify the cause of live video freezes by using the event timestamp.

[0065] In one or more embodiments of the present disclosure, the first indicator and / or the second indicator are analyzed to determine the cause of the freeze in the fused video to be analyzed, including: obtaining freeze events in the current live broadcast; counting the number of low speeds in which the network speed parameter is lower than the network speed threshold in the freeze event; if the number of low speeds is greater than or equal to the second number threshold, it is determined that the live video in the fused video is freezed.

[0066] In practical applications, in addition to using the aforementioned event timestamps to determine the cause of live video freezes, network speed can also be used to determine whether a live video freeze has occurred. Specifically, after a freeze event occurs, the network speed parameters within the time range of the freeze event are obtained. Furthermore, the number of times the network speed parameter falls below the network speed threshold during the current freeze event is counted. In other words, if the number of times the network speed parameter falls below the network speed threshold is greater than or equal to a second threshold, it is considered that a network problem is causing the live video freeze.

[0067] The network speed parameters mentioned here can be obtained using the player's built-in API (installed on a device that supports Virtual Hardware Statistics (VHS)) or the browser's Network API. When obtaining network speed parameters, trace back 5 seconds from the time the lag event is triggered and monitor 10 seconds backward, forming a 15-second analysis window. If the number of slow speed events within the window exceeds the second threshold (e.g., 5), it is determined to be a network issue.

[0068] Based on the above solution, we can see that by counting the number of low speeds in the time window where the freeze event occurs and combining it with the second number threshold for multi-indicator verification, we can reduce misjudgments and more accurately identify whether the live video freeze is caused by a network failure.

[0069] It's important to note that network performance parameter extraction methods vary depending on the operating system. If the client is an iOS (iPhone Operating System) device, since iOS devices don't support VHS, direct access to real-time network speed and bandwidth data is impossible. To reflect network speed, the request duration of the live stream's m3u8 file segments (ts (Transport Stream) files) is used as an alternative. Calculation steps: a. Use performance.getEntriesByType('resource') to retrieve performance data for all resource types, then filter the data to obtain tsRequests for all segmented TS files.

[0070] b. Filter tsRequests based on the camera's unique cameraKey identifier, extract the duration of each shard and add it to totalDuration, while also recording the number of shards, durationCount.

[0071] c. At the end of this timer loop, clearResourceTimings is executed to clear the performance resource area in order to avoid data overflow in the resource buffer (250 entries) and omission of TS shard resource information.

[0072] d. Multi-camera support: When monitoring multiple camera video streams, although the ts request of the videoPlayer can be distinguished by a unique identifier, clearing data using the clearResourceTimings method for each videoPlayer can affect resource data acquisition for other cameras. To address this issue, a singleton player class is implemented to ensure that all camera instances execute the clearResourceTimings method uniformly after acquiring performance data, preventing interference between cameras.

[0073] Through the above solution, the network speed information can be indirectly calculated using the request time of the TS segment file.

[0074] In one or more embodiments of the present disclosure, obtaining a second indicator for characterizing the stability of three-dimensional rendering from a three-dimensional rendering engine includes: in the three-dimensional rendering engine, driving a rendering loop through recursive calls; recording a timestamp using a callback function based on the rendering result; and calculating a second indicator representing the frame rate using the difference between the timestamps of two consecutive frames.

[0075] The rendering loop described here is implemented as follows: a recursive function calls itself within itself, forming a loop. For example, after the render() function completes the current frame, it actively calls requestAnimationFrame(render) to trigger the next frame. This infinite recursive call continuously executes the "update state → render frame" cycle, ensuring the continuity of animation or interaction.

[0076] In actual applications, the 3D rendering engine performs 3D rendering of live video in real time. For example, the timestamp parameter (type DOMHigh ResTimeStamp) obtained through the request Animation Frame callback can achieve microsecond accuracy (such as the accuracy of performance.now()), avoiding the errors caused by using Date.now() (millisecond accuracy).

[0077] The deltaTime difference between the timestamps of two consecutive frames reflects the rendering time of a single frame, which is the frame rate. For example, if deltaTime = 16ms, the theoretical frame rate is 60FPS (1000 / 16 ≈ 62.5FPS). If the calculated frame rate is 30FPS, it may be due to lag in the 3D rendering process.

[0078] Based on the above solution, when lag occurs, the 3D rendering FPS is further calculated. This allows analysis of whether lag occurred during the 3D rendering process. If lag occurs during the 3D rendering process, the fused video will be generated more slowly, which in turn will not meet the requirements for real-time playback of the fused video. This approach allows accurate calculation of the second indicator representing the 3D rendering effect.

[0079] In one or more embodiments of the present disclosure, the first indicator and / or the second indicator are analyzed to determine the cause of the freeze in the fused video to be analyzed, including: obtaining freeze events in the current live broadcast; counting the number of low FPS times in which the FPS is lower than the FPS threshold in the freeze event; if the number of low FPS times is greater than or equal to the third number threshold, it is determined that the three-dimensional rendering in the fused video is freezed.

[0080] In actual applications, we count the number of times the FPS falls below the FPS threshold within the time range corresponding to the current live broadcast freeze event. This is known as the low FPS count. If the low FPS count exceeds the third threshold, indicating that multiple FPS freezes occurred within a certain time range, we determine that the cause of the fused video freeze is freezes during the 3D rendering process.

[0081] This solution counts the number of low FPS occurrences during a freeze event and determines whether the freeze occurred during 3D rendering. The FPS here is obtained from the 3D rendering engine, and using this information allows for accurate judgment of the 3D rendering process. This improves the accuracy of freeze cause analysis for fused video.

[0082] In one alternative, after a live video is captured by a video capture device such as a camera, it is further sent to the client's 3D rendering engine. If the live video freezes, it will indirectly lead to a decrease in 3D rendering efficiency, that is, a decrease in FPS.

[0083] Therefore, when analyzing the cause of lag, if only the FPS decreases but the network speed or bandwidth doesn't, it indicates that 3D rendering is experiencing lag. If the FPS decreases while the network speed or bandwidth decreases, further analysis is needed to determine if the frequency of these decreases is consistent. If they are, it indicates that the fusion video lag is caused by a network issue. If only the network speed decreases but the FPS doesn't, it indicates that a problem with the live video network transmission is causing the fusion video playback lag.

[0084] In one or more embodiments of the present disclosure, obtaining the first indicator and the second indicator in the fused video includes: in response to a playback request of the fused video, obtaining the first indicator reported by the video player in the form of a network performance array when the fused video is played; obtaining the second indicator reported by the three-dimensional rendering engine in the form of a rendering performance array; clearing the network performance array and the rendering performance array after successful reporting; and terminating reporting after the live broadcast duration of the fused video exceeds a time threshold.

[0085] In actual applications, the client device receiving the live video can report the first indicator and the second indicator in real time as needed. When reporting, they can be reported in the form of a network performance array and a rendering performance array respectively.

[0086] For example, the configuration option enableLogVideoNetInfo provides the enableLogVideoNetInfo configuration property, which controls whether network performance monitoring is enabled, ensuring flexible monitoring. The aggregatedLogList network performance array, used to report information related to the first metric, stores network performance monitoring data. Its data structure is clear and organized: room name, camera unique identifier deviceId, and performance array metrics: metrics. Each record represents performance data at a 1-second timestamp, ensuring the time accuracy of data collection. The rendering performance array contains FPS-related data (timestamp, rendered frame count, etc.).

[0087] Initial data collection at playback startup: Clicking Play immediately triggers a network speed data report, providing timely access to initial playback performance information. This ensures rapid feedback on first-screen performance data and effectively avoids data loss during "cold starts." Reports are periodically updated during subsequent playback. Simply put, after playback starts, the scheduled reporter runs, reporting key data such as the current camera's basic information, network speed, and frame rate every 10 seconds. After each successful report, the aggregatedLogList is cleared to prepare space for the next round of data collection, ensuring consistently accurate and clear performance monitoring data.

[0088] When the live broadcast lasts for more than 15 minutes, the reporting will be automatically stopped and the timer will be cleared to avoid unnecessary resource usage due to long-term data collection.

[0089] This reporting mechanism, through multi-layered control with flexibility, precision, and efficiency, meets the needs of efficient real-time performance monitoring. Data reporting not only ensures real-time performance and integrity, but also provides intelligent support for system resource management. It achieves highly coordinated data collection, storage, and feedback, providing strong support for large-scale live stream monitoring.

[0090] In one or more embodiments of the present disclosure, corresponding stability measures are customized according to the cause of the lag, including: if the cause of the lag is determined to be live broadcast lag, the customized stabilization measures include: optimizing network configuration, increasing bandwidth and adjusting video bit rate at least one; if the cause of the lag is determined to be 3D rendering lag, the customized stabilization measures include: reducing model complexity and improving GPU performance at least one.

[0091] As mentioned above, after accurately determining the cause of the fusion video freeze through the solution described above, further stabilization measures can be formulated to improve the fusion video playback stability effect.

[0092] Specifically, if the lag is determined to be caused by the live video source, the following measures should be taken to stabilize the transmission: Network layer optimization: Deploy intelligent routing (such as HTTP / 3-based transmission protocols) and enable CDN dynamic acceleration to reduce cross-regional transmission latency. For weak network environments, adopt forward error correction (FEC) or redundant transmission strategies to reduce the impact of packet loss.

[0093] Adaptive bitrate adjustment: Dynamically switches the video stream resolution (e.g., from 1080p to 720p) or encoding parameters (e.g., H.265 instead of H.264) based on the real-time bandwidth, avoiding buffer exhaustion due to excessive bitrate.

[0094] Server-side QoS guarantee: Prioritizes the transmission of key frames (I frames), shortens the GOP interval to reduce decoding waiting time, and preloads the live stream through edge computing nodes to reduce the first frame loading delay.

[0095] If the lag is caused by insufficient 3D rendering performance, you need to optimize both hardware and content: Model and scene optimization: Reduce GPU load through LOD (level of detail) hierarchical loading, simplify high-polygon models (such as merging redundant meshes), reduce transparent materials and particle effects; spatially partition complex scenes (such as octree management) to render only visible areas.

[0096] Hardware resource optimization: Upgrade graphics card performance (such as higher video memory capacity), enable hardware-accelerated decoding (such as NVDEC / NVENC), or adopt a distributed cloud rendering solution to offload workloads on individual machines. For mobile devices, dynamically adjust rendering resolution and anti-aliasing levels to balance image quality and smoothness.

[0097] Asynchronous loading and resource management: Models, textures, and other resources are loaded asynchronously in chunks to avoid main thread blocking; object pooling technology is used to reuse rendering objects to reduce memory fragmentation and GC (garbage collection) stalls.

[0098] For ease of understanding, the following will be described through specific examples. Figure 2A schematic diagram illustrating the video analysis process of the present disclosure.

[0099] Create a video player on the client (here, assuming it's the user's mobile phone). Then, enable network performance monitoring and calculation for the live video. After registering the video player, initialize performance statistics-related properties. Start a timer (for example, 500ms) to begin calculating performance metrics. Calculate network performance metrics (such as network speed, network bandwidth, and lag time). Determine whether the statistical time exceeds the timer duration. Aggregate all network performance metrics within the timer duration. Use the get method to obtain network performance metrics for stability analysis of the live video.

[0100] During the fusion of live video, calculate the total frame time, minimum frame rate, maximum frame rate, maximum frame interval, minimum frame interval, average frame rate, and stable frame rate. From these calculation results, find the frame rate corresponding to the period of time when the fusion video freezes (this can be the average frame rate over a period of time) so that the FPS can be used to analyze the stability of 3D rendering.

[0101] When analyzing on the user side, start the network performance tracking reporting timer (for example, 10 seconds) and determine whether the total duration of the live fusion video exceeds 15 minutes. If so, terminate the reporting, stop the fusion video analysis, and clear the data. Otherwise, continue reporting until the 15th minute.

[0102] When determining a jam event, you can use the jam event to identify it. For example, when the video starts playing, get the difference between the current buffered time and the playback time: timeGap = videoPlayer.buffered().end(0) - videoPlayer.currentTime(). If timeGap ≤ 0, it indicates that the jam has begun, and the current time is recorded as the jam start time kadunStartTime. Calculation of continued jam: In the next timer loop, if timeGap ≤ 0, it indicates that the jam continues. Record the current jam duration lagTime = currentTime - kadunStartTime, and update kadunStartTime to currentTime. End of jam: If timeGap > 0 in the next timer loop, it indicates that the jam has ended. Record the current jam duration lagTime = currentTime - kadunStartTime, and reset kadunStartTime to -1 (indicating no jam). Accumulate all lagTimes within the aggregation timer to obtain the total jam duration of the time slice.

[0103] When performing real-time network speed statistics, check if the player videoPlayer.tech.vhs exists. If it exists, the number of bytes downloaded bytesLoaded = vhs.stats.mediaBytesTransferred can be obtained. The previous number of bytes downloaded previousBytesLoaded is initially 0. If bytesLoaded < previousBytesLoaded, the byte increment bytesDownloaded within the time slice is bytesLoaded; otherwise, bytesDownloaded is the byte difference bytesLoaded - previousBytesLoaded. The time interval timeElapsed = currentTime - previousTime, and the real-time download speed downloadSpeed = bytesDownloaded / 1024 / timeElapsed (unit: KB / s). Update previousTime and previousBytesLoaded to prepare for the next round of speed measurement.

[0104] When performing network bandwidth statistics, use vhs.stats.bandwidth to obtain the network bandwidth. Video playback requests video files in segments, and for each segment, an instant download speed value can be calculated through the segment size and download time, download speed = chunk size / download time. Combine the average speeds of multiple download tasks to estimate the bandwidth.

[0105] Based on any of the above embodiments, the present disclosure also provides a video analysis device. Figure 3 It is a structural schematic block diagram of a video analysis device according to an embodiment of the present disclosure. As Figure 3 shown, the video analysis device includes: a determination module 31 for determining the fusion video to be analyzed; wherein, the fusion video includes a live video with three-dimensional rendering effects.

[0106] An acquisition module 32 for acquiring a first index and a second index corresponding to the fusion video; wherein, the first index is used to characterize the stability index of the live video, and the second index is used to characterize the stability index of the three-dimensional rendering.

[0107] An analysis module 33 for analyzing the first index and / or the second index to determine the cause of stuttering of the fusion video. <0000​​​The acquisition module 32 is used to obtain a first indicator for characterizing the stability of the live video from the video player; and obtain a second indicator for characterizing the stability of the three-dimensional rendering from the three-dimensional rendering engine.

[0110] The acquisition module 32 is used to obtain the playback freeze event and the event timestamp corresponding to the freeze event from the video player through a specified interface or network layer; and obtain the first indicator based on the live broadcast freeze event and the event timestamp.

[0111] The analysis module 33 is used to determine the cache time and playback time of the live video based on the event timestamp; calculate the difference between the cache time and the playback time as the delay time; count the number of freezes in the live video playback process with a delay time less than zero; if the number of freezes is greater than or equal to the first count threshold, it is determined that the live video has freezes.

[0112] The analysis module 33 is used to obtain the freeze event in the current live broadcast; count the number of low speeds in which the network speed parameter is lower than the network speed threshold during the freeze event; if the number of low speeds is greater than or equal to the second number threshold, it is determined that the live broadcast video in the fused video is freezed.

[0113] The acquisition module 32 is used to drive the rendering loop through recursive calls in the 3D rendering engine; record the timestamp using a callback function according to the rendering result; and calculate the second indicator representing the frame rate using the difference between the timestamps of two consecutive frames.

[0114] The analysis module 33 is used to obtain the freeze event in the current live broadcast; count the number of low FPS times when the FPS is lower than the FPS threshold in the freeze event; if the number of low FPS times is greater than or equal to the third number threshold, it is determined that the three-dimensional rendering in the fused video is freezed.

[0115] The acquisition module 32 is used to respond to the playback request of the fusion video, obtain the first indicator reported by the video player in the form of a network performance array when the fusion video is played; obtain the second indicator reported by the three-dimensional rendering engine in the form of a rendering performance array; clear the network performance array and the rendering performance array after the report is successful; and terminate the reporting after the live broadcast duration of the fusion video exceeds the time threshold.

[0116] Customization module 34 is used to customize stabilization measures including at least one of optimizing network configuration, increasing bandwidth, and adjusting video bit rate if the cause of the lag is determined to be live broadcast lag; if the cause of the lag is determined to be 3D rendering lag, then the customized stabilization measures include at least one of reducing model complexity and improving GPU performance.

[0117] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0118] The execution entity of the image synthesis method and the image synthesis model training method in the specific embodiments of the present disclosure can be an electronic device such as a server (including a local server or a cloud server).

[0119] Therefore, based on any of the above embodiments, the present disclosure also provides an electronic device that can execute the image synthesis method and image synthesis model training method of any of the embodiments described above in the present disclosure.

[0120] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation entrances shall be provided for users to choose to authorize or refuse.

[0121] Figure 4 The present invention is a block diagram showing the structure of an electronic device according to an embodiment of the present invention.

[0122] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. The bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0123] Bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component Architecture (EISA) bus. Buses can be classified as address buses, data buses, control buses, and the like. For ease of illustration, this figure shows only one connecting line, but this does not imply that there is only one bus or only one type of bus.

[0124] The present disclosure also provides a readable storage medium having a computer program stored therein, which is used to implement the above-mentioned method when the computer program is executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0125] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the process or function of the present disclosure is performed in whole or in part.

[0126] A computer program or instruction can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instruction can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any accessible medium or a data storage device such as a server or data center that integrates one or more accessible media. The accessible medium can be a magnetic medium such as a floppy disk, hard disk, or magnetic tape; an optical medium such as a digital video disk; or a semiconductor medium such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or can include both volatile and non-volatile types of storage media.

[0127] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing method device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing method device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0131] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in a suitable manner in any one or more embodiments / methods or examples. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and the features of different embodiments / methods or examples, unless they are contradictory.

[0132] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0133] Those skilled in the art will appreciate that the above embodiments are merely intended to clearly illustrate the present disclosure and are not intended to limit the scope of the present disclosure. Other changes or modifications may be made based on the above disclosure, and such changes or modifications are still within the scope of the present disclosure.

Claims

1. A video analysis method, characterized in that: The method comprises: Determining a fused video to be analyzed; wherein the fused video includes a live video with a three-dimensional rendering effect; Obtaining a first indicator and a second indicator corresponding to the fused video; wherein the first indicator is used to represent a stability indicator of the live video, and the second indicator is used to represent a stability indicator of the three-dimensional rendering; Analyze the first indicator and / or the second indicator to determine a cause of freeze in the fused video; Customize corresponding stability measures based on the cause of the lag.

2. The method according to claim 1, characterized in that The obtaining of the first indicator and the second indicator in the fused video includes: Obtaining a first indicator for characterizing the stability of a live video from a video player; A second indicator for characterizing the stability of the three-dimensional rendering is obtained from the three-dimensional rendering engine.

3. The method according to claim 2, characterized in that The obtaining, from the video player, a first indicator for characterizing the stability of the live video includes: Obtain playback freeze events and corresponding event timestamps from the video player through a specified interface or network layer. The first indicator is obtained based on the live broadcast freeze event and the event timestamp.

4. The method according to claim 3, characterized in that The analyzing the first indicator and / or the second indicator to determine the cause of the freeze of the fused video to be analyzed includes: Determining a buffering time and a playback time of the live video according to the event timestamp; Calculating the difference between the buffering time and the playback time as the delay time; Counting the number of freezes during live video playback where the delay time is less than zero; If the number of freezes is greater than or equal to the first threshold value, it is determined that freezes occur in the live video.

5. The method according to claim 2, characterized in that The analyzing the first indicator and / or the second indicator to determine the cause of the freeze of the fused video to be analyzed includes: Get the freeze event in the current live broadcast; Count the number of times the network speed parameter is lower than the network speed threshold during the freeze event; If the number of low speeds is greater than or equal to the second number threshold, it is determined that the live video in the fused video is stuck.

6. The method according to claim 2, characterized in that The obtaining of a second indicator for characterizing the stability of the three-dimensional rendering from the three-dimensional rendering engine includes: In the three-dimensional rendering engine, a rendering loop is driven by recursive calls; According to the rendering results, use the callback function to record the timestamp; The second indicator representing the frame rate is calculated using the time stamp difference between two consecutive frames.

7. The method according to claim 6, characterized in that The analyzing the first indicator and / or the second indicator to determine the cause of the freeze of the fused video to be analyzed includes: Get the freeze event in the current live broadcast; Count the number of low FPS times during the freeze event when the FPS is lower than the FPS threshold; If the number of low FPS times is greater than or equal to a third threshold, it is determined that a freeze occurs in the three-dimensional rendering of the fused video.

8. The method according to claim 2, characterized in that The obtaining of the first index and the second index corresponding to the fused video includes: In response to a playback request for a fused video, obtaining the first indicator reported by the video player in the form of a network performance array when the fused video is played; Obtaining the second indicator reported by the three-dimensional rendering engine in a rendering performance array; After the report is successful, the network performance array and the rendering performance array are cleared; After the duration of the fusion video live broadcast exceeds the time threshold, the reporting is terminated.

9. The method according to claim 1, characterized in that The corresponding stability measures are customized according to the cause of the jamming, including: If it is determined that the cause of the freeze is live streaming freeze, then customizing the stabilization measures includes: optimizing network configuration, increasing bandwidth, and adjusting video bit rate; If it is determined that the cause of the jamming is 3D rendering jamming, then customizing the stabilization measures includes: reducing model complexity and improving GPU performance.

10. An electronic device, characterized in that: include: a memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 9.

Images