Multi-screen video processing method and device, equipment, storage medium and program product

Abstract

The invention discloses a multi-screen video processing method and device, equipment, a storage medium and a program product. The method comprises the following steps: in a multi-screen playing mode, determining a target image acquisition device needing to pre-pull streams in image acquisition devices corresponding to a plurality of split screens in a current split screen group, determining a pre-pull stream duration for the target image acquisition device, and enabling one split screen to correspond to one image acquisition device; a stream pushing instruction is issued to the target image acquisition equipment, so that the target image acquisition equipment pushes streams to a streaming media server; and pulling and caching the streaming data of the target image acquisition equipment through the connection between the terminal equipment and the streaming media server. According to the embodiment of the invention, the problem of multi-screen playing delay can be effectively solved. And the watching experience of the user is improved, smooth playing of the video during multi-screen switching is guaranteed, and the technical effects of saving server resources and enhancing the practicability and market competitiveness of the security and protection system are achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of video processing, and particularly relates to a multi-screen video processing method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] In the prior art, there are many problems in related video processing technologies. For example, some solutions achieve second-opening playback through pre-pull streaming, but do not consider network environment differences, which easily causes waste of bandwidth resources and poor pre-pull streaming effect in poor network conditions; some solutions depend on user pull streaming requests to start cache processing, which causes high playback delay when the network or server is under heavy pressure; and some solutions pre-pull streaming for all devices, which causes a large amount of unnecessary data transmission and increases server load and network bandwidth consumption.

[0003] Therefore, how to solve the multi-screen video playback delay has become a technical problem to be solved at present. SUMMARY

[0004] Embodiments of the present application provide a multi-screen video processing method, device, equipment, storage medium and program product, which can effectively solve the problem of multi-screen playback delay, improve user viewing experience, ensure smooth video playback during multi-screen switching, save server resources, and enhance the practicality and market competitiveness of the security system.

[0005] In a first aspect, embodiments of the present application provide a multi-screen video processing method applied to a terminal device, including: in a multi-screen playback mode, determining a target image acquisition device that needs to be pre-pulled in a plurality of image acquisition devices corresponding to a plurality of split screens in a current split screen group, determining a pre-pull streaming time length for the target image acquisition device, wherein one split screen corresponds to one image acquisition device; issuing a push streaming instruction to the target image acquisition device to make the target image acquisition device push streaming to a streaming media server; and pulling and caching streaming data of the target image acquisition device through a connection between the terminal device and the streaming media server.

[0006] In a possible implementation, the target image acquisition device includes at least one of: an image acquisition device corresponding to a previous split screen group adjacent to the current split screen group in a playback order, an image acquisition device corresponding to a next split screen group adjacent to the current split screen group in the playback order, an image acquisition device whose alarm quantity reaches a preset attention condition, and an image acquisition device in a preset white list.

[0007] In a possible implementation, the determining of the pre-pull streaming time length for the target image acquisition device includes: in a case where a network environment of the terminal device is a WI-FI environment, determining the pre-pull streaming time length as a first time length; and in a case where the network environment is a data network environment, determining the pre-pull streaming time length as a second time length, the first time length being greater than the second time length.

[0008] In a possible implementation, after the step of determining the pre-pull duration of the target image capture device, the method further includes: when entering the multi-screen playing page, calling the pre-cached stream data of the target image capture device for playing.

[0009] In a possible implementation, the method further includes: setting a target button as a trigger entry at a specified position of the multi-screen playing page; displaying at least one thumbnail of a split-screen group in a target area of the multi-screen playing page according to the trigger entry; and playing the selected split-screen group as a current split-screen group in response to a selection operation of the at least one thumbnail of the split-screen group.

[0010] In a possible implementation, before the multi-screen playing mode, the method further includes: in a multi-screen preview mode, obtaining a list of image capture devices corresponding to a plurality of split screens in a current split-screen group, where one split screen corresponds to one image capture device; periodically obtaining a current captured screen of the image capture device corresponding to each split screen, and generating a thumbnail of the image capture device based on the current captured screen; and in each split screen of a playing list page, updating a cover picture of the image capture device corresponding to the split screen based on the periodically generated thumbnail.

[0011] In a possible implementation, the method further includes: in a case of entering a playing mode, sending a playing request to the stream media server, and stopping the steps of periodically sending a trigger instruction to the stream media server and generating the thumbnail; and in a case of exiting the playing list page, displaying a last frame of the screen as a cover picture of the target image capture device on the next time of entering the playing list page.

[0012] In a possible implementation, after the step of pulling and caching the stream data of the target image capture device, the method further includes: after entering the multi-screen playing mode, initializing a multi-screen preview list interface; pulling alarm data of the image capture device corresponding to the multi-screen preview list interface, and counting the number of alarms of each image capture device; and superimposing and displaying the number of alarms of each image capture device in the thumbnail of the corresponding split-screen group in the multi-screen preview list interface.

[0013] In a second aspect, the embodiments of the present application provide a multi-screen video processing device, the device comprising: a determination module configured to determine, in a multi-screen playing mode, a target image acquisition device that needs to be pre-pulled in a plurality of image acquisition devices corresponding to a plurality of split screens in a current split screen group, and determine a pre-pull time length for the target image acquisition device, wherein one split screen corresponds to one image acquisition device; a push module configured to issue a push instruction to the target image acquisition device to make the target image acquisition device push to a streaming media server; and a pre-pull module configured to pull and cache streaming data of the target image acquisition device through a connection between the terminal device and the streaming media server.

[0014] In a third aspect, the embodiments of the present application provide a multi-screen video processing device, the device comprising: a processor and a memory storing computer program instructions; and the processor executes the computer program instructions to implement the multi-screen video processing method according to any one of the first aspect.

[0015] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, the computer readable storage medium storing computer program instructions, and the computer program instructions are executed by a processor to implement the multi-screen video processing method according to any one of the first aspect.

[0016] In a fifth aspect, the embodiments of the present application provide a computer program product, and instructions in the computer program product are executed by a processor of an electronic device to make the electronic device execute the multi-screen video processing method according to any one of the first aspect.

[0017] The embodiments of the present application provide a multi-screen video processing method, device, equipment, storage medium and program product. By determining, in a multi-screen playing mode, a target image acquisition device that needs to be pre-pulled in a plurality of image acquisition devices corresponding to a plurality of split screens in a current split screen group, determining a pre-pull time length, issuing a push instruction to the target image acquisition device to make the target image acquisition device push to a streaming media server, connecting the terminal device with the streaming media server, and then pulling and caching streaming data of the target image acquisition device. The pre-pull processing in multi-screen playing can effectively solve the problems of camera first screen playing delay and multi-screen playing delay. The user's viewing experience is improved, the video is played smoothly during multi-screen switching, the server resources are saved, and the practicality and market competitiveness of the security system are enhanced. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. For those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 is a flowchart of a multi-screen video processing method provided by an embodiment of the present application; Figure 2 is a display structure diagram of adjacent split-screen groups provided by an embodiment of the present application; Figure 3 is a flowchart of a method for determining a pre-pull stream duration provided by an embodiment of the present application; Figure 4 is a flowchart of another multi-screen video processing method provided by an embodiment of the present application; Figure 5 is a flowchart of still another multi-screen video processing method provided by an embodiment of the present application; Figure 6 is a flowchart of yet another multi-screen video processing method provided by an embodiment of the present application; Figure 7 is a flowchart of a method for dynamically updating a thumbnail provided by an embodiment of the present application; Figure 8a is an effect diagram of a target button for displaying a thumbnail by a terminal provided by an embodiment of the present application; Figure 8b is another effect diagram of a target button for displaying a thumbnail by a terminal provided by an embodiment of the present application; Figure 8c is still another effect diagram of a target button for displaying a thumbnail by a terminal provided by an embodiment of the present application; Figure 9a is an effect diagram of a method for calculating a pixel size provided by an embodiment of the present application; Figure 9b is another effect diagram of a method for calculating a pixel size provided by an embodiment of the present application; Figure 10 is a flowchart of a method for counting a pre-pull stream alarm number provided by an embodiment of the present application; Figure 11 is an effect diagram of a method for counting a pre-pull stream alarm number provided by an embodiment of the present application; Figure 12 is a diagram for calculating saved time provided by an embodiment of the present application; Figure 13 is a flowchart of another method for counting a pre-pull stream alarm number provided by an embodiment of the present application; Figure 14 is a flowchart of still another multi-screen video processing method provided by an embodiment of the present application; Figure 15 is a structural diagram of a multi-screen video processing device provided by an embodiment of the present application; Figure 16This is a schematic diagram of the structure of a multi-screen video processing device provided in an embodiment of this application. Detailed Implementation

[0020] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0021] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0022] Video pre-buffering refers to a technique where the client downloads part or all of the video data from the server to its local buffer before the video is actually played, aiming to reduce stuttering and latency during playback. Its core principle is to optimize the user experience through preloading strategies, with particularly significant effects in scenarios with unstable networks or high bitrates.

[0023] In security systems, each camera is connected to a display screen, and multiple cameras are set up at various angles to achieve a multi-screen display effect. The cameras capture video images in real time, and the captured images are transmitted to the display device via a server to achieve real-time display.

[0024] However, when the camera is first turned on, the system takes a considerable amount of time to load when a user requests to view the camera feed, resulting in a continuous "loading" message on the screen. This initial delay significantly impacts the user experience. This delay is primarily due to network transmission latency. Data transmission from the camera to the user's device requires traversing the network, and uncertainties in network conditions (such as bandwidth limitations and network congestion) can increase packet transmission time. Server processing latency also occurs; after receiving a user request, the server needs to process the request and begin pushing the video stream, which introduces a delay. Finally, client decoding latency is present; even after the video stream reaches the client, the player needs to decode it before displaying it, a process that also takes time.

[0025] Meanwhile, latency also exists in multi-screen playback scenarios. For example, in large venues such as supermarkets and shopping malls, multiple cameras are deployed to achieve comprehensive monitoring. For ease of management and viewing, users often need to view the footage from multiple cameras simultaneously via multi-screen (e.g., a four-screen split view). Since the video stream from each camera needs to be loaded independently, when a user opens a multi-screen view, the footage from all cameras needs to be loaded one by one, which leads to significant latency and affects the user experience. Furthermore, due to update delays in the multiple camera thumbnails, the cover image seen by the user is not the latest live stream, impacting the user experience.

[0026] To address the aforementioned latency issues with cameras, this application provides a multi-screen video processing method, apparatus, device, storage medium, and program product. In multi-screen playback mode, the pre-streaming duration of the target image acquisition device that needs to be pre-streamed among the multiple image acquisition devices corresponding to the current split-screen group is determined. The stream data of the target image acquisition device is then retrieved and cached according to the set pre-streaming duration. This pre-streaming processing effectively solves the problems of latency in both the initial screen playback and multi-screen playback during multi-screen playback. It improves the user viewing experience, ensures smooth video playback during multi-screen switching, saves server resources, and enhances the practicality and market competitiveness of security systems. The multi-screen video processing method provided in this application is described below.

[0027] Figure 1 A flowchart illustrating a multi-screen video processing method provided in an embodiment of this application is shown. Figure 1 As shown, the steps of the multi-screen video processing method specifically include: S101~S103.

[0028] S101. In multi-screen playback mode, determine the target image acquisition device that needs to be pre-streamed among the multiple image acquisition devices corresponding to the current split screen group, and determine the pre-stream duration for the target image acquisition device. Here, one split screen corresponds to one image acquisition device.

[0029] The image acquisition device mentioned here can be understood as a device with image acquisition capabilities, such as a camera or industrial camera. The current split-screen group contains multiple split screens, each containing multiple display screens, and each display screen is connected to a corresponding camera.

[0030] The target image acquisition devices mentioned here include at least one of the following: the image acquisition device corresponding to the previous split-screen group in the playback order adjacent to the current split-screen group, the image acquisition device corresponding to the next split-screen group in the playback order adjacent to the current split-screen group, the image acquisition device whose alarm count reaches the preset attention condition, and the image acquisition devices in the preset whitelist.

[0031] The adjacent playback order of the split-screen groups mentioned here can be understood as follows: based on the display device selecting a split-screen group for display, the display split-screen groups set adjacent to the current split-screen group can also be selected for display. Figure 2 This is a display structure diagram of an adjacent split-screen group provided in an embodiment of this application. (Reference) Figure 2 The structure consists of three adjacent split-screen groups on the display screen. The current split-screen group contains four split-screens, the previous split-screen group contains four split-screens, and the next split-screen group contains four split-screens.

[0032] The image acquisition devices in the preset whitelist mentioned here can be understood as whitelisted devices that the user has added for special attention, such as those not in the device list, those set by the client, or those configured by the platform.

[0033] Furthermore, in multi-screen playback mode, multiple split-screen groups are displayed on the screen, with fixed settings between adjacent split-screen groups. To improve the streaming data caching effect, it is necessary to first determine the target image acquisition device that needs to be pre-streamed, i.e., to find the target camera. After selecting the camera to be pre-streamed, to avoid continuously pre-streaming the camera and wasting resources, it is necessary to set the pre-streaming duration. By appropriately setting the pre-streaming duration, it is ensured that the initial display of the camera is not affected by latency.

[0034] S102. Send a push stream command to the target image acquisition device so that the target image acquisition device pushes the stream to the streaming media server.

[0035] The streaming media server mentioned here can be understood as a server used to connect the target image acquisition device and the terminal display device.

[0036] After selecting the target image acquisition device and the pre-stream duration, the pre-stream operation needs to be initiated. A push command is sent from the client to the target image acquisition device, containing information about the target image acquisition device and the pre-stream duration. This connects the target image acquisition device to the streaming media server, providing a source for acquiring the pre-stream data in the next step.

[0037] S103. By connecting the terminal device to the streaming media server, retrieve and cache the streaming data of the target image acquisition device.

[0038] The term "terminal device" here can be understood as either a mobile terminal device or a non-mobile terminal device. Examples include mobile phones, tablets, or other devices with display functions.

[0039] After the streaming media server is connected to the terminal device, the streaming media server transmits the pre-pulled stream data obtained from the push stream to the terminal device for display.

[0040] The client sends a streaming command to the signaling platform. When the signaling platform receives the streaming request from the client, it sends a streaming command to the device that is determined to need to stream. The client establishes a connection with the streaming media server and obtains streaming data by dynamically caching data (Group of Pictures, GOP). When the user enters the playback page, the cached streaming data GOP is retrieved for playback.

[0041] In one possible scenario, in multi-screen playback mode, the system pre-fetches data streams from adjacent split screens and dynamically adjusts the pre-fetching duration based on the network environment. By pre-acquiring the stream address and fetching the data without decoding or rendering, the system allows users to directly play the cached stream data when switching between split screens, achieving a near-instantaneous playback effect. This effectively resolves the latency issue caused by the initial camera activation.

[0042] This application provides a multi-screen video processing method. In multi-screen playback mode, it identifies the target image acquisition device that needs pre-streaming among the multiple image acquisition devices corresponding to the current split-screen group, determines the pre-streaming duration, and then sends a push-stream command to the target image acquisition device to push the stream to the streaming media server. This establishes a connection between the terminal device and the streaming media server, thereby pulling and caching the stream data from the target image acquisition device. By implementing pre-streaming processing during multi-screen playback, it effectively solves the problems of latency in first-screen and multi-screen playback. This improves the user viewing experience, ensures smooth video playback during multi-screen switching, and achieves the technical effects of saving server resources and enhancing the practicality and market competitiveness of security systems.

[0043] Furthermore, in step S101, the process of determining the pre-flow duration... Figure 3This is a schematic diagram of a process for determining the pre-pull flow duration provided in an embodiment of this application. Figure 3 This description is based on the previous embodiment. According to... Figure 2 The provided diagram shows the specific steps for determining the pre-flow duration, including: S301~S302.

[0044] S301. When the network environment of the terminal device is a WI-FI environment, the pre-pulling time is determined to be the first time.

[0045] S302. In the case of a data network environment, the pre-pulling duration is determined to be the second duration, and the first duration is longer than the second duration.

[0046] The network environment mentioned here can be understood as the network connection type of the terminal device. This includes Wi-Fi wireless network environment, wired network environment, or mobile data network environment.

[0047] Furthermore, when obtaining pre-stream data for multi-screen playback from terminal devices, different pre-stream durations are set based on the network stability of the provided network environment. In the case of a Wi-Fi or wired LAN environment, indicating a stable network, a longer pre-stream duration can be set, thus determining the first pre-stream duration. In the case of a mobile data network environment, to avoid wasting additional client bandwidth, the pre-stream duration needs to be shortened, resulting in a second duration. By setting the pre-stream duration in the Wi-Fi environment to be longer than that in the data network environment, while ensuring the cached pre-stream data, multi-screen video playback is cached while reducing bandwidth waste.

[0048] For example, if the user's terminal device is in a Wi-Fi environment, the pre-streaming duration is set to 45 seconds. If the user is in a mobile data network environment, the pre-streaming duration is set to 5 seconds.

[0049] The method for determining the pre-stream duration in this application embodiment sets different pre-stream durations according to the different network environments of the terminal devices. This reduces traffic waste and improves the pre-stream response effect while ensuring that multi-screen video playback is not affected by delays.

[0050] After setting the pre-stream duration in step S101, when entering the multi-screen playback page, the pre-cached stream data of the target image acquisition device is retrieved for playback.

[0051] Pre-streaming involves pre-fetching the stream address and pre-caching the stream data without decoding or rendering the data. This ensures that the pre-fetched stream cache data is available during the initial camera startup period, resolving the image buffering and waiting issue caused by latency during the initial camera startup.

[0052] In one possible example scenario, Figure 4 This is a flowchart illustrating another multi-screen video processing method provided in an embodiment of this application. Figure 4 The description is based on the above embodiments. Figure 4 The provided illustration shows the specific steps of the multi-screen video processing method, including: 1. When a user enters the playlist page, the system determines whether the device needs to pre-pull the stream. If not, no action is taken.

[0053] 2. If the device needs to pre-pull the flow, then determine the user's network environment.

[0054] 3. If the user is in a Wi-Fi environment, set the pre-stream duration to 45 seconds.

[0055] 4. If the user is in a data network environment, set the pre-pull duration to 5 seconds.

[0056] 5. The client sends a pull command to the signaling platform.

[0057] 6. When the signaling platform receives a streaming request from the client, it sends a streaming push command to the device that is determined to need to stream.

[0058] 7. The client establishes a connection with the streaming media server, starts pre-pull streaming, and pushes the stream from the camera to the terminal device.

[0059] 8. Dynamically cache GOPs based on the environment.

[0060] 9. When a user enters the playback page, retrieve the cached GOP for playback.

[0061] This application provides a multi-screen video processing method that, before a user initiates a playback request, pre-fetches and caches Group of Pictures (GOPs) to a local server and dynamically adjusts the number of cached GOPs based on the network environment. When a user initiates a playback request, since most of the required data has already been prepared and stored on the local server, a "second-open" experience for the camera feed can be achieved, greatly shortening the first screen loading time. This effectively solves the problem of long loading times caused by network transmission latency, server processing latency, and player decoding latency, significantly improving the user's viewing experience.

[0062] Before the multi-screen playback mode, when a user entered the playback page, the platform interface was called to retrieve or fetch the last frame of the previous live stream before playback began. However, calling the platform interface could result in latency due to network issues, and fetching the last frame meant the cover image displayed to the user wouldn't be the latest live stream time. Both methods negatively impacted the user experience. To address this latency issue...Figure 5 This is a flowchart illustrating another multi-screen video processing method provided in an embodiment of this application. According to... Figure 5 The provided diagram illustrates the specific steps of the multi-screen video processing method, including S501~S503.

[0063] S501. In multi-screen preview mode, obtain a list of image acquisition devices corresponding to multiple split screens in the current split screen group, where one split screen corresponds to one image acquisition device.

[0064] S502: Periodically acquire the images currently captured by the image acquisition devices corresponding to multiple split screens, and generate thumbnails of the image acquisition devices based on the currently captured images.

[0065] In multi-screen preview mode, users can access the playlist page through the client interface, which displays a list of available devices and their corresponding cover images. Trigger commands are periodically sent to the server to capture the live feed from the camera. Upon receiving the client's trigger command, the server checks if the interval between the current time and the generation of the previous live image is less than the timer period. If the interval is less than the timer period, the server directly returns the previously generated image to the client without re-capturing from the camera. If the interval is greater than the timer period, the server continues capturing the current camera feed and generating a corresponding thumbnail as the cover image.

[0066] S503. In each split screen of the playlist page, based on the thumbnail generated periodically, update and display the cover image of the image acquisition device corresponding to that split screen.

[0067] Based on the periodically generated camera thumbnails, the server establishes a communication connection with the corresponding camera using the device information provided by the camera, acquires the latest video frames, generates a real-time image, and obtains the thumbnail. This process of acquiring thumbnails is repeated continuously with periodic triggers, constantly updating the corresponding cover image. The client receives the thumbnails returned by the server at the intervals specified by the periodic trigger command. After receiving the thumbnail, the client renders it and displays it in the corresponding device's cover image position on the playlist page, replacing the previous old cover image. This allows for a clear view of the camera's current real-time status, enabling dynamic updates of cover images for multiple cameras and presenting an instant visual effect.

[0068] This application embodiment ensures the cover image is updated in real time by periodically requesting real-time images from the image acquisition device to generate thumbnails in multi-screen preview mode. Clicking the thumbnail allows for quick switching of the current playback device, enabling smooth interactive operation.

[0069] In one possible example scenario, based on the updated cover image, according to Figure 5The provided diagram illustrates the specific steps of the multi-screen video processing method, including S504~S505.

[0070] S504. When entering playback mode, the steps of sending a playback request to the streaming media server and stopping the periodic sending of trigger commands and thumbnail generation to the streaming media server.

[0071] When a user taps the split-screen button to enter playback mode, the client sends a playback request and simultaneously stops the currently scheduled triggering command for that image acquisition device. Upon receiving the stop triggering command, the server stops generating thumbnails for that image acquisition device.

[0072] S505. When exiting the playlist page, use the last frame displayed as the cover image for the target image acquisition device when it enters the playlist page again.

[0073] If the user exits the playlist page, the last frame of the video stream at that moment will be saved as the cover image for the camera to be displayed on the playlist page later, making it easier to quickly identify the camera next time.

[0074] In one possible example scenario, Figure 6 This is a flowchart illustrating another multi-screen video processing method provided in an embodiment of this application. According to... Figure 6 The provided illustration shows the specific steps of the multi-screen video processing method, including: After the user enters the playlist page on the client interface, a list of available devices and their corresponding cover images are displayed. The system checks if the device is online; if offline, a default offline background image is displayed. If online, subsequent steps are executed. A built-in timer in the client starts working, sending a trigger command to the platform at preset intervals (e.g., every 15 seconds), informing the platform that it needs to obtain the device's current live feed. Upon receiving the client's trigger command, the platform checks if the interval between the current time and the generation of the previous live image is less than 15 seconds. If less than 15 seconds, the platform directly returns the previously generated cover image to the client, skipping step 5 and proceeding to step 6. If greater than 15 seconds, subsequent steps are executed. Based on the camera information in the command, the platform establishes a communication connection with the corresponding camera and obtains the latest... A new video frame generates a real-time image, and this process is repeated continuously with timed triggers, constantly updating the real-time image content. The client receives real-time image data returned by the platform according to the time interval of sending trigger commands. After receiving the real-time image, the client immediately renders it and displays it on the cover image position of the corresponding device on the playlist page, replacing the previous old cover image, intuitively showing the current real-time status of the device, realizing dynamic updates of cover images for multiple devices, and presenting an instant visual effect. When the user clicks to enter the split-screen playback mode, the client sends a playback request and stops the current timed trigger command for that camera. After receiving the stop trigger command, the platform stops generating real-time images for that camera. If the user exits the playlist page, the last frame of the video stream at that moment is saved as the cover image displayed for that camera on the playlist page later, facilitating quick device identification next time.

[0075] The multi-screen video processing method provided in this application pre-triggers the platform to generate a live image before the user starts playback, and stores it using a continuously refreshed single-image method. The client also periodically retrieves this live image to update the thumbnail. Once pre-streaming stops or the user starts playback, the generation of the live image immediately stops. This method ensures the real-time nature and freshness of the cover image, improving the user's viewing experience. Simultaneously, controlling the timing of live image generation through timed triggering effectively saves resource consumption.

[0076] To further illustrate the thumbnail generation process, Figure 7 This is a flowchart illustrating a method for dynamically updating thumbnails according to an embodiment of this application. Figure 7 The specific steps of the dynamic updating method for the provided illustration and thumbnail include: S701~S703.

[0077] S701. Set a target button as the trigger entry point at a specified location on the multi-screen playback page.

[0078] S702. Display at least one thumbnail of a split-screen group in the target area of ​​the multi-screen playback page based on the trigger entry.

[0079] For example, in the case of a four-screen split-screen setup, an "Expand Details" button can be placed in the upper right corner of the four-screen live stream page as a trigger entry point. After the user clicks it, a specific area of ​​the page will be presented in a neat four-grid layout, specifically for displaying thumbnails of other devices within the organization, providing users with more device preview options.

[0080] In one possible example scenario, Figure 8a This is an illustration of the effect of a target button displaying a thumbnail on a terminal, provided in an embodiment of this application. Figure 8b This is an illustration of the effect of a target button for displaying thumbnails on a terminal, as provided in another embodiment of this application. According to... Figure 8a As shown in Figure 8b, the target button can be set randomly at a specified location as the target area, without being limited by the specified location.

[0081] S703, In response to the received selection operation of at least one split-screen group thumbnail, the selected split-screen group is played as the current split-screen group.

[0082] Thumbnail generation relies on a backend data processing workflow and is linked to the acquisition of real-time images from the device. Real-time images are captured from the corresponding camera through timed or on-demand triggering, and then processed through cropping, compression, and format adaptation to convert them into thumbnail sizes and formats suitable for multi-screen display, ensuring image clarity and rapid loading.

[0083] In one possible example scenario, Figure 8c This is an illustration of the effect of a target button displaying a thumbnail on a terminal, as provided in another embodiment of this application. According to... Figure 8c The provided illustrations, in the case of a four-screen multi-view setup, show the arrangement of each four-screen segment according to a pre-defined relationship. The target button can be customized, allowing users to choose whether it's displayed in the top left, top right, bottom left, or bottom right corner of the playback page. Users can also choose between horizontal and vertical thumbnail display. For example, when the user selects the top right corner for horizontal arrangement (Figure 8a), and the top left corner for vertical arrangement (Figure 8b), the user experience is greatly enhanced. Each group of thumbnails in the multiple four-screen segments has interactive attributes. By binding click events, when a user clicks on a group of thumbnails, the device in the currently playing camera list page is replaced, seamlessly transitioning to a new playback. If the camera goes offline during preview, the camera's cover image is changed to the offline background image; if the camera is unbound or its sharing is canceled, the corresponding camera is removed from the preview frame. This achieves a dynamic update effect for the thumbnail preview.

[0084] Furthermore, the size of the thumbnail displayed on the terminal screen is obtained in the following manner.

[0085] First, let's assume the phone screen resolution is... That is, the horizontal pixel count of the terminal device's display screen is x Vertical pixels are y For a four-screen layout, each horizontally divided screen occupies half the screen width, i.e. x / 2 pixels. Each split screen occupies half the screen height vertically, that is... y / 2 pixels. For example, if the resolution of the terminal device is 1920×1080, then the resolution of each split screen is 960×540.

[0086] Secondly Figure 9a This is a rendering of a pixel size calculation provided in an embodiment of this application. According to... Figure 9a The provided illustration assumes that the horizontal direction of each of the four screens in the thumbnail is... a 1 pixel, with a spacing of 1 pixel from the edge. l If there are 100 pixels, then we get Equations 1 and 2: Equation 1 Equation 2 Regarding the process of calculating pixel size, Figure 9b This is an example of another calculation of pixel size provided in an embodiment of this application. According to... Figure 9b The provided illustrations assume that the vertical direction of each of the four screens in the thumbnail is... b 1 pixel, with a spacing of 1 pixel from the edge. m If there are 100 pixels, then we get Equations 3 and 4: Equation 3 Equation 4 To ensure consistency with the display ratio of the terminal device, the aspect ratio of the pixels is limited, resulting in Equation 5: Equation 5 According to Equation 5, for example, if the phone resolution is 1920×1080, the horizontal screen split and the edge spacing are... l =40 pixels, vertical screen split and edge spacing m =140 pixels, therefore a ≈293 pixels b =135 pixels; different mobile phones have different screen sizes, resolutions, and pixel densities, so calculations need to be made based on the specific parameters of each phone. In practical applications, the system also needs to fine-tune the screen display to ensure the aesthetics of the interface and the ease of operation.

[0087] The thumbnail dynamic update method provided in this application generates thumbnails by periodically acquiring real-time images captured by the camera and replacing the camera's cover image, thereby achieving real-time updates of the cover image and solving the problem of camera thumbnail update latency.

[0088] Even after addressing the latency issues caused by multi-screen preview and playback, latency problems still exist in multi-screen playback scenarios. In large venues such as supermarkets and shopping malls, multiple cameras are typically deployed for comprehensive monitoring. For ease of management and viewing, it's often necessary to simultaneously view the footage from multiple cameras using a multi-screen setup (e.g., a four-screen split). However, since each camera's video stream needs to be loaded independently, when a user opens a multi-screen view, all camera feeds need to be loaded sequentially, leading to latency and impacting the user experience. Figure 10 This is a schematic flowchart of a pre-pull flow alarm quantity statistics method provided in an embodiment of this application. According to... Figure 10 The provided diagram shows the steps of the pre-pull flow alarm number statistics method, specifically including: S1001~S1003.

[0089] S1001. After entering the multi-screen playback mode, initialize the multi-screen preview list interface.

[0090] S1002, Retrieve alarm data of the corresponding image acquisition device from the multi-screen preview list interface, and count the number of alarms for each image acquisition device.

[0091] After a user enters the multi-screen playback mode, the multi-screen preview screen is initialized. Then, the alarm data for each camera in the preview list is retrieved from the server, and the number of alarms for each camera is calculated in real time. This provides reference data for the next step of alarm display.

[0092] S1003. Display the alarm count of each image acquisition device in the thumbnail of the corresponding split-screen group in the multi-screen preview list interface.

[0093] To provide a notification effect, the alarm count for each image acquisition device is displayed in a superimposed thumbnail of the corresponding split-screen group in the multi-screen preview list interface. In one possible example scenario... Figure 11 This is an example of an alarm count display provided in this application. As shown in Figure 11, the number 5 in the preview box represents the real-time alarm count of the top left screen in the second split-screen group; the number 8 represents the real-time alarm count of the top right screen in the third split-screen group.

[0094] Furthermore, by browsing the preview interface and using the thumbnails displayed, users can quickly identify cameras with a relatively large number of alarms, thereby determining the abnormal display issues of the cameras that require special attention.

[0095] The pre-stream alarm count method provided in this application pre-loads the video streams from a list of cameras to be played before the user initiates a playback request. During playback, the alarm count for each camera is counted in real time and sorted according to the alarm count. The four cameras with the highest pre-stream alarm counts are prioritized, and the areas covered by these cameras are designated as key areas of focus. This facilitates the display of abnormal situations.

[0096] Furthermore, by counting the number of alarms, we can activate the cameras of key interest, saving time when accessing the playback interface. To further save time... Figure 12 This is a schematic diagram illustrating a calculation of time savings provided in an embodiment of this application. According to... Figure 12 The provided diagram illustrates this. Let the total time required to open the live stream originally be T, where the streaming time is t1 and the decoding / rendering time is t2, thus T = t1 + t2. After adopting the pre-streaming method, the time required to open the live stream becomes T'. Since pre-streaming saves t1 time, T' = t2, leading to the time saving expressed in Equation 6: Equation 6 After entering the playback interface of the camera you are focusing on, you can view the camera's live feed and detailed alarm information to further understand and handle the alarm situation.

[0097] In one possible example scenario, Figure 13 This is a schematic diagram of another pre-pull flow alarm quantity statistics method provided in an embodiment of this application. According to Figure 13 The provided diagram illustrates the steps involved in the pre-pull flow alarm count method, which specifically include: S131, The user enters multi-screen playback mode; S132. Initialize the multi-screen preview screen; S133, Retrieve alarm data from cameras in the preview list; S134. Real-time statistics and updates of camera alarm data; S135. Display the number of alarms overlaid in the preview box; S136. The user identifies a camera with a large number of alarms; S137. The user selects and focuses on the camera; S138. Enter the playback interface of the camera; S139. View the camera's live feed and alarm information.

[0098] The pre-stream alarm count method provided in this application, after the user enters the multi-screen preview mode, counts the alarm data of the cameras in the preview list in real time, and displays the alarm count overlaid in the preview box to achieve priority sorting of alarm data. Combined with pre-stream technology, it prioritizes pre-streaming alarms from key cameras, helping users quickly locate and handle alarm issues.

[0099] By pre-loading video streams from the list of cameras to be played, and counting the number of alarms for each camera in real time during playback, the system prioritizes pre-loading the video streams from the four cameras with the highest number of alarms. During multi-screen playback, by pre-loading the video streams from the previous and next split screens, low-latency multi-screen playback is achieved, significantly improving the user experience and solving the latency problem in multi-screen playback scenarios.

[0100] In one possible example scenario, Figure 14 This is a flowchart illustrating another multi-screen video processing method provided in an embodiment of this application. According to... Figure 14 The provided illustration shows that when a user enters a multi-screen playback scenario, a multi-channel video preview scheme, a multi-screen pre-streaming scheme, and a thumbnail alarm overlay scheme are activated. When the multi-channel video preview scheme is activated, thumbnails of the remaining cameras in the current organization are acquired. The thumbnail data is periodically refreshed by the platform and dynamically changed based on the cameras' online / offline status. When the thumbnail alarm overlay scheme is activated, alarm count statistics are enabled, and the alarm count is overlaid onto the camera cover. Alarm data is acquired from the platform in real time, and the alarm count is dynamically updated. When the multi-screen pre-streaming scheme is activated, in addition to pre-streaming the data from the previous and next split screens, the top four cameras with the highest alarm counts must also be pre-streamed. During pre-streaming, the stream address and stream data are acquired in advance, but no decoding or rendering is performed. This application's embodiment solves the problem of quickly switching between screens for multi-channel video playback, focusing on optimizing the user's operational experience when watching multiple videos on a terminal, fully meeting the user's needs for quick switching and instant viewing of different split-screen videos. During the current split-screen playback phase, by pre-streaming the data from the previous and next split screens, the technical obstacle of slow image loading when switching from the current split screen to the next split screen is cleverly overcome. This effectively ensures that users can quickly load images when switching split screens, greatly improving the smoothness and timeliness of viewing and bringing users a more convenient and efficient video viewing experience.

[0101] Figure 15 This is a schematic diagram of the structure of a multi-screen video processing device provided in an embodiment of this application. According to... Figure 15 The provided diagram shows that the structure of the multi-screen video processing device 100 specifically includes: The determination module 151 is used to determine the target image acquisition device that needs to be pre-streamed among the multiple image acquisition devices corresponding to the current split screen in the multi-screen playback mode, and to determine the pre-streaming duration for the target image acquisition device, wherein one split screen corresponds to one image acquisition device; The streaming module 152 is used to send streaming instructions to the target image acquisition device so that the target image acquisition device can stream to the streaming media server. The pre-pull module 153 is used to pull and cache the streaming data of the target image acquisition device through the connection between the terminal device and the streaming media server.

[0102] The multi-screen video processing device provided in this embodiment can be as follows: Figure 15 The multi-screen video processing device 100 shown can realize Figures 1 to 14 For a detailed description of the technical effects of the multi-screen video processing method shown, please refer to [link / reference]. Figures 1 to 14 The corresponding explanation is concise and will not be elaborated upon here.

[0103] Figure 16 A schematic diagram of the hardware structure for multi-screen video processing provided in an embodiment of this application is shown. The multi-screen video processing device may include a processor 1601 and a memory 1602 storing computer program instructions.

[0104] Specifically, the processor 1601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0105] Memory 1602 may include mass storage for data or instructions. For example, and not limitingly, memory 1602 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1602 may include removable or non-removable (or fixed) media. Where appropriate, memory 1602 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 1602 is non-volatile solid-state memory.

[0106] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to the first aspect of this disclosure.

[0107] The processor 1601 reads and executes computer program instructions stored in the memory 1602 to implement any of the multi-screen video processing methods in the above embodiments.

[0108] In one example, the multi-screen video processing device may further include a communication interface 1603 and a bus 1610. For example, Figure 16 As shown, the processor 1601, memory 1602, and communication interface 1603 are connected through bus 1610 and complete communication with each other.

[0109] The communication interface 1603 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0110] Bus 1610 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1610 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0111] Furthermore, in conjunction with the multi-screen video processing methods described in the above embodiments, this application embodiment can provide a computer storage medium for implementation. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the multi-screen video processing methods described in the above embodiments.

[0112] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the multi-screen video processing methods described in the above embodiments.

[0113] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0114] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0115] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0116] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0117] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A multi-screen video processing method, applied to a terminal device, characterized in that, The method comprises the following steps: In the multi-screen playing mode, a target image acquisition device needing pre-pull streaming among a plurality of image acquisition devices corresponding to a plurality of split screens in a current split screen group is determined, and a pre-pull streaming duration for the target image acquisition device is determined; A push streaming instruction is issued to the target image acquisition device, so that the target image acquisition device pushes streaming to a streaming media server; Through the connection between the terminal device and the streaming media server, streaming data of the target image acquisition device is pulled and cached.

2. The method of claim 1, wherein, The target image acquisition device comprises at least one of an image acquisition device corresponding to a previous split screen group adjacent to the current playing split screen group in the playing order, an image acquisition device corresponding to a next split screen group adjacent to the current split screen group in the playing order, an image acquisition device whose alarm quantity reaches a preset attention condition, and an image acquisition device in a preset white list.

3. The method of claim 1, wherein, The pre-pull streaming duration for the target image acquisition device is determined, comprising: In the case that the network environment of the terminal device is a WI-FI environment, the pre-pull streaming duration is determined as a first duration; In the case that the network environment is a data network environment, the pre-pull streaming duration is determined as a second duration, and the first duration is greater than the second duration.

4. The method of claim 1, wherein, After the pre-pull streaming duration for the target image acquisition device is determined, the method further comprises: When entering a multi-screen playing page, pre-cached streaming data of the target image acquisition device is played.

5. The method of claim 4, wherein, The method further comprises: A target button is set as a trigger entry at a specified position of the multi-screen playing page; At least one thumbnail of a split screen group is displayed in a target area of the multi-screen playing page according to the trigger entry; In response to a selected operation of the at least one thumbnail of the split screen group, the selected split screen group is played as a current split screen group.

6. The method of claim 1, wherein, Before the multi-screen playing mode, the method further comprises: In a multi-screen preview mode, a list of image acquisition devices corresponding to a plurality of split screens in a current split screen group is obtained, wherein one split screen corresponds to one image acquisition device; Timely generated thumbnails of the image acquisition devices are generated based on pictures currently collected by the image acquisition devices; In each split screen of a playing list page, a cover picture of an image acquisition device corresponding to the split screen is updated based on the timely generated thumbnails.

7. The method of claim 6, wherein, The method further comprises: In the case of entering a playing mode, a playing request is sent to the streaming media server, and the steps of timely sending a trigger instruction to the streaming media server and generating the thumbnails are stopped; In the case of exiting the playing list page, a last frame of displayed picture is used as a cover picture of the target image acquisition device on the next time of entering the playing list page.

8. The method of claim 1, wherein, After the streaming data of the target image acquisition device is pulled and cached, the method further comprises: After entering the multi-screen playing mode, a multi-screen preview list interface is initialized; Alarm data of image acquisition devices corresponding to the multi-screen preview list interface is pulled, and alarm quantities of each image acquisition device are counted; The alarm quantity of each image acquisition device is superimposed and displayed in the thumbnail of the corresponding split screen group in the multi-screen preview list interface.

9. A multi-screen video processing apparatus, characterized by comprising: The device comprises: A determination module is configured to determine a target image acquisition device that needs to be pre-pulled in a plurality of split screens in a current split screen group in a multi-screen playing mode, and determine a pre-pulling time length for the target image acquisition device, wherein one split screen corresponds to one image acquisition device; A pushing module is configured to issue a pushing instruction to the target image acquisition device, so as to push the target image acquisition device to a streaming media server; A pre-pulling module is configured to pull and cache streaming data of the target image acquisition device through a connection between a terminal device and the streaming media server.

10. A multi-screen video processing device, comprising: The device comprises a processor and a memory storing computer program instructions; The processor executes the computer program instructions to implement the multi-screen video processing method of any one of claims 1-8.

11. A computer readable storage medium, characterized in that, The computer program instructions are stored on the computer readable storage medium, and the computer program instructions are executed by the processor to implement the multi-screen video processing method of any one of claims 1-8.

12. A computer program product, characterised in that, The instructions in the computer program product are executed by the processor of the electronic device, so that the electronic device executes the multi-screen video processing method of any one of claims 1-8.

Images