Display method and device of multiple display pictures, storage medium and electronic device

Abstract

The invention discloses a multi-display-picture display method and device, a storage medium and an electronic device, and relates to the technical field of smart home, and the method comprises the steps: receiving original video stream data from different image collection devices through a network transmission channel; decoding the original video stream data to obtain video frame data, and obtaining preloaded video frame data obtained after preloading the video frame data; determining first video frame data corresponding to a first operation of the target object from the preloaded video frame data, and determining second video frame data corresponding to a second operation of the target object from the preloaded video frame data; and displaying the first video frame data on the first display page, and displaying the second video frame data on the second display page, thereby solving the technical problem of poor display effect of multiple display pictures.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and more specifically, to a method, device, storage medium and electronic device for displaying multiple display screens. Background Art

[0002] In modern multimedia applications, multiple display screens are generally presented in the form of picture-in-picture and floating windows, providing users with the convenience of viewing multiple screens simultaneously on the main screen. However, the functions of picture-in-picture (PiP) and floating windows often rely on complex graphical user interface programming and additional software support. This implementation method not only has a high technical threshold, but also requires real-time rendering of images. When displaying multiple screens in a low-speed network environment or on a low-performance device with limited resources, there is often a display delay, which seriously affects the smooth playback of video content. Especially in application scenarios that require real-time interaction such as video conferencing, it seriously affects the user experience, the display effect of the picture is not good, and limits the application of these functions in low-performance devices and low-speed network environments.

[0003] Therefore, in the related art, there is a technical problem that the display effect of multiple display screens is not good.

[0004] In view of the technical problem of poor display effect of multiple display images in related technologies, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method, device, storage medium and electronic device for displaying multiple display screens, so as to at least solve the technical problem of poor display effect of multiple display screens in the related art.

[0006] According to an embodiment of the present application, a method for displaying multiple display screens is provided, comprising: receiving original video stream data from different image acquisition devices through a network transmission channel, wherein the network transmission channel represents a channel for transmitting the original video stream data between a terminal device and the different image acquisition devices; decoding the original video stream data to obtain video frame data, and obtaining preloaded video frame data obtained after preloading the video frame data; determining first video frame data corresponding to a first operation of a target object from the preloaded video frame data, and determining second video frame data corresponding to a second operation of the target object from the preloaded video frame data, wherein the first operation represents selecting a first display page for displaying the first video frame data, and the second operation represents selecting a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in a display area of ​​the same terminal device, and the display format of the first display page and the display format of the second display page support mutual switching; displaying the first video frame data on the first display page, and displaying the second video frame data on the second display page.

[0007] In an exemplary embodiment, the original video stream data is decoded to obtain video frame data, including: grouping the original video stream data according to the channel number of the network transmission channel to obtain multiple groups of video stream data to be decoded, wherein each group of video stream data to be decoded corresponds to an independent network transmission channel; and decoding the multiple groups of video stream data to be decoded using video decoders respectively to obtain decoded video frame data.

[0008] In an exemplary embodiment, after determining the first video frame data corresponding to the first operation of the target object from the preloaded video frame data, and determining the second video frame data corresponding to the second operation of the target object from the preloaded video frame data, the method further includes: upon receiving a switching operation of the target object on the display area, generating a first control instruction based on the switching operation; and using the first control instruction to switch the display format of the first display page and the display format of the second display page.

[0009] In an exemplary embodiment, using the first control instruction to switch the display format of the first display page and the display format of the second display page includes: storing the current frame data in the first video frame data to the first buffer, and preloading the next frame data of the first video frame data to the second buffer; using the first control instruction to switch the display format of the first display page and the display format of the second display page, and at the same time reading the next frame data of the first video frame data from the second buffer, and displaying the next frame data of the first video frame data on the first display page.

[0010] In an exemplary embodiment, after displaying the first video frame data on the first display page and displaying the second video frame data on the second display page, the method further includes: acquiring a first timestamp of the first video frame data and a second timestamp of the second video frame data according to a preset period; upon determining that a timestamp difference between the first timestamp and the second timestamp is greater than a preset difference, pausing display of a next frame of the first video frame data on the first display page if a playback time corresponding to the first timestamp is earlier than a playback time corresponding to the second timestamp; or, accelerating display of a next frame of the second video frame data on the second display page if a playback time corresponding to the first timestamp is earlier than a playback time corresponding to the second timestamp.

[0011] In an exemplary embodiment, after displaying the first video frame data on the first display page and displaying the second video frame data on the second display page, the method further includes: in case of monitoring that the terminal device generates abnormal video stream data due to a playback failure, executing a reconnection instruction, re-establishing a channel for transmitting the video stream data between the terminal device and the different image acquisition devices, and re-receiving the video stream data; in case of determining that the re-received video stream data is displayed normally in the display area, determining the generation time of the abnormal video stream data, and determining the display time of the first frame data from the re-received video stream data; and updating the re-received video stream data based on the comparison result of the display time and the generation time.

[0012] In an exemplary embodiment, the re-received video stream data is updated based on the comparison result of the display moment and the generation moment, including: in a case where it is determined that the display moment is later than the generation moment, determining a first time period between the display moment and the generation moment, and obtaining third video stream data belonging to the first time period from the original video stream data; and updating the re-received video stream data based on the third video stream data.

[0013] According to another aspect of an embodiment of the present application, a display device for multiple display screens is also provided, including: a receiving module, used to receive original video stream data from different image acquisition devices through a network transmission channel, wherein the network transmission channel represents a channel for transmitting the original video stream data between a terminal device and the different image acquisition devices; a preloading module, used to decode the original video stream data to obtain video frame data, and obtain preloaded video frame data obtained after preloading the video frame data; a determination module, used to determine first video frame data corresponding to a first operation of a target object from the preloaded video frame data, and determine second video frame data corresponding to a second operation of the target object from the preloaded video frame data, wherein the first operation represents selecting a first display page for displaying the first video frame data, and the second operation represents selecting a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in a display area of ​​the same terminal device, and the display format of the first display page and the display format of the second display page support mutual switching; a display module, used to display the first video frame data on the first display page, and to display the second video frame data on the second display page.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for presenting multiple display screens when running.

[0015] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the multi-display screen display method through the computer program.

[0016] In an embodiment of the present application, raw video stream data from different image acquisition devices is received through a network transmission channel, wherein the network transmission channel represents a channel for transmitting the raw video stream data between a terminal device and the different image acquisition devices; the raw video stream data is decoded to obtain video frame data, and preloaded video frame data obtained after preloading the video frame data is obtained; first video frame data corresponding to a first operation of a target object is determined from the preloaded video frame data, and second video frame data corresponding to a second operation of the target object is determined from the preloaded video frame data, wherein the first operation represents selecting a first display page for displaying the first video frame data, and the second operation represents selecting a first display page for displaying the first video frame data, and the second operation represents selecting a first display page for displaying the first video frame data. The second display interface for displaying the second video frame data is selected, wherein the first display page and the second display page are located in the display area of ​​the same terminal device, and the display format of the first display page and the display format of the second display page support mutual switching; the first video frame data is displayed on the first display page, and the second video frame data is displayed on the second display page; by adaptively optimizing the videos collected by different cameras, the stability of video transmission is enhanced, and even in the case of large network fluctuations, the present application can still provide a stable video picture, and display multiple display pages in the terminal device according to user operations, thereby improving the display flexibility and efficiency of multiple display pages, and improving the adaptability of display pages and user operations. The above technical solution solves the technical problem of poor display effect of multiple display pictures, thereby improving the display effect and user experience of multiple display pictures. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a schematic diagram of the hardware environment of a method for displaying multiple display screens according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a method for displaying multiple display screens according to an embodiment of the present application;

[0021] Figure 3 is a flowchart of a method for displaying multiple display screens according to an embodiment of the present application;

[0022] Figure 4 It is a structural block diagram of a multi-display screen display device according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, the present application, a product or an apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0025] According to one aspect of an embodiment of the present application, a method for displaying multiple display screens is provided. The method for displaying multiple display screens is widely used in smart home (Smart Home), smart home, smart home device ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the above-mentioned method for displaying multiple display screens can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0026] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0027] In this embodiment, a method for displaying multiple display images is provided, which is applied to the above terminal device. Figure 2 : is a flowchart of a method for displaying multiple display screens according to an embodiment of the present application, and the process includes the following steps:

[0028] Step S202, receiving raw video stream data from different image acquisition devices through a network transmission channel, wherein the network transmission channel refers to a channel for transmitting the raw video stream data between a terminal device and the different image acquisition devices;

[0029] Step S204, decoding the original video stream data to obtain video frame data, and obtaining preloaded video frame data obtained by preloading the video frame data;

[0030] Step S206, determining first video frame data corresponding to a first operation of the target object from the preloaded video frame data, and determining second video frame data corresponding to a second operation of the target object from the preloaded video frame data, wherein the first operation represents selection of a first display page for displaying the first video frame data, and the second operation represents selection of a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in a display area of ​​the same terminal device, and a display format of the first display page and a display format of the second display page support mutual switching;

[0031] Step S208: display the first video frame data on the first display page, and display the second video frame data on the second display page.

[0032] Through the above steps, original video stream data from different image acquisition devices is received through a network transmission channel, wherein the network transmission channel represents a channel for transmitting the original video stream data between a terminal device and the different image acquisition devices; the original video stream data is decoded to obtain video frame data, and preloaded video frame data obtained after preloading the video frame data is obtained; first video frame data corresponding to a first operation of a target object is determined from the preloaded video frame data, and second video frame data corresponding to a second operation of the target object is determined from the preloaded video frame data, wherein the first operation represents selecting a first display page for displaying the first video frame data, and the second operation represents Select a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in the display area of ​​the same terminal device, and the display format of the first display page and the display format of the second display page support mutual switching; display the first video frame data on the first display page, and display the second video frame data on the second display page; by adaptively optimizing the videos collected by different cameras, the stability of video transmission is enhanced, and even in the case of large network fluctuations, the present application can still provide a stable video picture, and display multiple display pages in the terminal device according to user operations, thereby improving the display flexibility and efficiency of multiple display pages, and improving the adaptability of display pages and user operations. The above technical solution solves the technical problem of poor display effect of multiple display pictures, thereby improving the display effect and user experience of multiple display pictures.

[0033] Optionally, in one embodiment, the first display page is, for example, a main screen display page representing a display area of ​​a terminal device, and the second display interface can be understood as a secondary screen display page for realizing picture-in-picture and floating window effects. Picture-in-picture means that a smaller second display interface is displayed in the first display page, providing a multi-view function, allowing users to view multiple video sources at the same time. Floating window means that an independent, draggable second display interface is displayed on top of the first display page, and auxiliary information or additional functions can be synchronously displayed when the user freely moves the floating window. Such as floating toolbars, chat windows, etc.

[0034] This embodiment supports synchronization of the picture frame rate of each camera in multiple display modes of the display area, so that the pictures remain synchronized when the user switches the display mode.

[0035] Among them, the display area supports multiple display modes, and the display formats supported by the display interfaces in different display modes are different. For example, in the single-channel full-screen display mode, only one camera screen occupies the entire screen, and the display format is full-screen display. In the dual-screen display mode, the two camera screens are displayed side by side or arranged up and down, and the display format is side by side. In the picture-in-picture display mode, referring to the above content, a smaller second display interface is displayed in the first display page. At this time, the display format of the first display page is full-screen display, and the display format of the second display interface is small window display.

[0036] In an exemplary embodiment, the original video stream data is decoded to obtain video frame data, and the implementation process may include: grouping the original video stream data according to the channel number of the network transmission channel to obtain multiple groups of video stream data to be decoded, wherein each group of video stream data to be decoded corresponds to an independent network transmission channel; respectively decoding the multiple groups of video stream data to be decoded using a video decoder to obtain decoded video frame data. This embodiment optimizes the video stream processing process by preloading video frame data, can adapt to different network conditions, improves the reliability of data transmission, solves the problem of delay and asynchrony in multi-camera screen switching, and improves the smoothness of video playback.

[0037] Furthermore, this embodiment can obtain the preloaded video frame data obtained after preloading the video frame data, and then perform a rendering operation on the preloaded video frame data to improve the viewing experience of video playback.

[0038] In an exemplary embodiment, after determining the first video frame data corresponding to the first operation of the target object from the preloaded video frame data, and determining the second video frame data corresponding to the second operation of the target object from the preloaded video frame data, further, when a switching operation of the target object on the display area is received, a first control instruction is generated based on the switching operation; and the first control instruction is used to switch the display format of the first display page and the display format of the second display page.

[0039] It is understandable that in order to achieve seamless screen switching, this application preloads the video streams of all connected cameras. Even if the user is currently viewing only one screen, the video streams of other cameras are still decoded synchronously in the background and kept updated synchronously. In this way, when the user switches the screen, he only needs to switch the rendering focus from one video frame to another loaded video frame on the front end, without re-establishing the connection or re-requesting the video stream. This mode of pre-loading and running in the background can reduce excessive consumption of device resources, improve the performance and response speed of terminal equipment operation, extend the battery life of the device, and improve the overall stability of operation.

[0040] This application supports continuous running of video monitoring in the background, and reduces resource consumption through low power consumption mode, which solves the power consumption problem of the device during the monitoring process, while ensuring the continuous monitoring function running in the background. Even when performing other tasks, you can still continue to view the video screen through the floating window, which not only extends the battery life of the device, but also enhances the durability and practicality of monitoring.

[0041] The switching operation can be triggered by gestures or buttons. For example, users can switch from the current monitoring screen to the next monitoring screen by sliding gestures. Different switching operations meet the personalized needs of users. Users can freely adjust the size, position, transparency and other attributes of the floating window according to their personal needs, thereby improving the richness of user operation options and operational flexibility.

[0042] Optionally, this application also supports users to click the camera icon on the interface to directly switch to the video stream of the specified camera, to achieve monitoring of specific areas, and better meet different monitoring needs.

[0043] Optionally, in one embodiment, a second control instruction can be generated based on the first operation, and the second control instruction can be used to determine the first video frame data corresponding to the first operation of the target object from the preloaded video frame data; a third control instruction can be generated based on the second operation, and the third control instruction can be used to determine the second video frame data corresponding to the second operation of the target object from the preloaded video frame data.

[0044] Optionally, in one embodiment, when multiple video streams are displayed simultaneously, if a user clicks on the thumbnail of a certain video, the rendering layer priority of the thumbnail is changed in the background based on the user's click to enlarge the thumbnail to display on the main screen, and the user interaction layout is adjusted on the main screen.

[0045] In an exemplary embodiment, the process of using the first control instruction to switch the display format of the first display page and the display format of the second display page is implemented by the following steps: storing the current frame data in the first video frame data to the first buffer, and preloading the next frame data of the first video frame data to the second buffer; using the first control instruction to switch the display format of the first display page and the display format of the second display page, and at the same time reading the next frame data of the first video frame data from the second buffer, and displaying the next frame data of the first video frame data on the first display page.

[0046] Optionally, in one embodiment, the current frame data in the second video frame data is stored in the first buffer, and the next frame data of the second video frame data is preloaded into the second buffer; the display format of the first display page and the display format of the second display page are switched using the third control instruction, and the next frame data of the second video frame data is read from the second buffer, and the next frame data of the second video frame data is displayed on the second display page.

[0047] It should be noted that the present application adopts a double buffering mechanism (Double Buffering), which can reduce the situation of screen freezes during switching. Specifically, at the moment of switching, the present embodiment saves the currently displayed video frame in the front buffer, and preloads the next video frame to the back buffer. After the switch is triggered, the front buffer and the back buffer are exchanged, and the user sees the decoded and rendered video frame. Alternatively, before the user triggers the switching operation, the first few frames of video are pre-loaded for the next camera screen. In this way, when the user clicks to switch, the new picture is immediately presented, and there will be no temporary black screen, which improves the user's switching experience.

[0048] In an exemplary embodiment, after displaying the first video frame data on the first display page and displaying the second video frame data on the second display page, further, a first timestamp of the first video frame data and a second timestamp of the second video frame data may be collected according to a preset period; when it is determined that the timestamp difference between the first timestamp and the second timestamp is greater than the preset difference, if the playback time corresponding to the first timestamp is earlier than the playback time corresponding to the second timestamp, then the display of the next frame data of the first video frame data on the first display page is paused; or, if the playback time corresponding to the first timestamp is earlier than the playback time corresponding to the second timestamp, then the display of the next frame data of the second video frame data on the second display page is accelerated.

[0049] This embodiment improves the synchronization of multiple display screens by adding a timestamp to each video stream. Specifically, by comparing the timestamps of each video stream, it is ensured that multiple screens can maintain synchronous display of images at the same time point. In this way, in view of the shortcomings that current remote monitoring can only display the screen of one camera at a time, and there may be delays or freezes, this application realizes seamless switching and synchronous updating of multi-camera screens by performing timestamp synchronization and multi-threaded processing, improves the transmission efficiency of multiple video streams, and can monitor multiple camera screens at the same time, greatly improving monitoring efficiency and continuity.

[0050] Optionally, if the playback time corresponding to the first timestamp is later than the playback time corresponding to the second timestamp, the display of the next frame data of the second video frame data on the second display page is paused, or the display of the next frame data of the first video frame data on the first display page is accelerated.

[0051] Since the network transmission delays of different cameras may vary, this application will perform delay correction on each video stream. For example, the process implemented using the frame synchronization algorithm is: periodically check the timestamps of all video streams to ensure that the frames displayed by each video stream are at the same time point. If a video channel loses frames or has excessive delays due to network problems, suspend the update of the video channel, and wait for the video stream to be restored before performing time synchronization and resuming playback. By comparing the timestamps of the camera video streams, the transmission delay of each video channel is calculated. If there is a large delay in a video channel, fine-tune the playback of other videos (such as temporarily slowing down or speeding up the video playback) so that each video screen can be synchronized to the same time point.

[0052] The timestamp may be determined based on the acquisition time or transmission time of the camera.

[0053] In an exemplary embodiment, after displaying the first video frame data on the first display page and the second video frame data on the second display page, further, when it is monitored that the terminal device generates abnormal video stream data due to a playback failure, a reconnection instruction is executed to re-establish a channel for transmitting the video stream data between the terminal device and the different image acquisition devices, and the video stream data is received again; when it is determined that the re-received video stream data is displayed normally in the display area, the generation time of the abnormal video stream data is determined, and the display time of the first frame data is determined from the re-received video stream data; and the re-received video stream data is updated based on the comparison result of the display time and the generation time.

[0054] This embodiment can prompt the user at the front end when a video stream is interrupted due to a network or device failure, and try to reconnect the camera. If the connection is restored successfully, the video screen will be automatically updated and synchronized to the time point of other current video streams. In the case of large network delay, the frame rate or resolution of the video can also be temporarily reduced to ensure the continuity of the video and avoid video freezes or increased delays due to high network fluctuations.

[0055] In an exemplary embodiment, a technical solution is further proposed for updating the re-received video stream data based on the comparison result of the display moment and the generation moment, specifically including: when it is determined that the display moment is later than the generation moment, determining a first time period between the display moment and the generation moment, and obtaining third video stream data belonging to the first time period from the original video stream data; and updating the re-received video stream data based on the third video stream data.

[0056] Alternatively, when it is determined that abnormal video stream data is generated due to abnormal display of the first display page or abnormal display of the second display page, a restart instruction is executed to receive the video stream data again.

[0057] In order to better understand the process of the method for displaying multiple display screens, the implementation method flow of the method for displaying multiple display screens is described below in combination with an optional embodiment, but it is not intended to limit the technical solution of the embodiment of the present application.

[0058] In this embodiment, a method for displaying multiple display images is provided. Figure 3 is a flowchart of a method for displaying multiple display screens according to an embodiment of the present application, such as Figure 3 As shown, the specific steps are as follows:

[0059] Step S301: Use a remote camera to collect real-time video stream and transmit it to the terminal;

[0060] The video stream data collected by the camera can be transmitted to the terminal device through a wireless network (such as Wi-Fi, 4G / 5G or local area network). The application in the terminal device distinguishes each video source through the unique identifier of the camera (such as camera ID or IP address) and establishes an independent network connection for each camera video stream. Among them, the unique identifier of the above camera can be understood as the channel number used to distinguish each video source, and the independent network connection established for each camera video stream is equivalent to the above independent network transmission channel.

[0061] For example, video data may be transmitted via RTSP (Real-Time Streaming Protocol) to achieve low-delay and high-quality transmission of video stream data.

[0062] Step S302: receiving and decoding the video stream at the mobile terminal;

[0063] After the video data reaches the terminal device, it is decoded by a hardware decoder or a software decoder. For example, using hardware accelerated decoding can ensure smooth decoding of multiple video streams without affecting device performance.

[0064] Step S303: After decoding each video independently, display them synchronously according to the timestamp;

[0065] Specific terminal devices can use a multi-threaded or asynchronous processing model for independent decoding to ensure that each video stream is processed in an independent thread to avoid resource competition or delay accumulation.

[0066] Step S304: Render the video frame using the GPU and display it on the screen;

[0067] Based on the above steps, the decoded video frames are rendered to the screen through the GPU. Multiple rendering layers can be used to display the images from different cameras, ensuring that each image can be rendered at the same time without image overlap or flickering.

[0068] Step S305: Implement the surface in the picture and the floating window.

[0069] Optionally, referring to the above steps, a screen display system is also proposed, which specifically includes a camera module, an image processing module, an image synthesis module, and an interface display module. This system uses a camera module to capture camera image data in real time and output the original image data stream to the image processing module. The image processing module can store the captured image data in the image data buffer for subsequent processing, so that the image data is temporarily stored to ensure the stability and efficiency of data transmission. Then, the image data is pre-processed such as denoising and color correction to improve the image quality and reduce the difficulty of subsequent processing. Among them, denoising refers to removing noise in the image and improving the clarity of the image. Color correction can adjust the color of the image to more realistically reflect the scene. The pre-processed image data is further processed, including image operations such as scaling, cropping and synthesis.

[0070] Among them, scaling can adjust the size of the image to meet the display requirements of the main image and the floating window. Cropping refers to extracting a specific area from the image for the floating window effect. Combining can combine multiple image data into a complete image to prepare for subsequent display.

[0071] The image synthesis module then processes the image processed by the image processing module into the final main image and floating window image. By embedding the floating window image in the main image, a picture-in-picture effect is formed. The transparency and position of the floating window image are then processed to ensure that the floating window image is correctly displayed above the main image.

[0072] Finally, the interface display module displays the final synthesized image, including the main image and the floating window image. For example, the entire camera image is the main view, and the floating window image is displayed on the main view to provide a picture-in-picture effect. At the same time, visual operation controls can also be provided to allow users to interact with the image, such as allowing users to adjust the position and size of the floating window and set the image display effect.

[0073] During this process, the resource management module can be used to perform data encryption and access control.

[0074] Through the above embodiments, efficient image processing technology and resource management strategies are used to improve the stability and fluency of video playback, reduce the performance burden of the device, and improve the stability of the overall operation. By optimizing the image processing algorithm and reducing the dependence on the graphical user interface library, the resource consumption of this application is reduced, and the performance on resource-constrained devices such as mobile phones is improved. Thus, the camera image can be displayed smoothly even on low-performance devices, improving the user experience.

[0075] Based on the above content, optionally, in one embodiment, the picture-in-picture effect can be achieved by combining the following process. The specific steps are as follows:

[0076] Step 1, create two UIViews, named largeVideoView and smallVideoView, which are used as containers for large and small screens. The initial layout of these two UIViews is: the large screen fills the entire parent view, and the small screen is fixed in the lower right corner.

[0077] Among them, the large screen is, for example, the above-mentioned first display interface, and the small screen is, for example, the second display interface.

[0078] Step 2, use NSLayoutConstraint to define the initial positions of the two UIViews. The large screen fills the entire screen, and the small screen is located in the lower right corner. The size of the small screen is defined as 30% of the parent view through the multipliers of widthAnchor and heightAnchor.

[0079] Step 3, add a tap gesture to the two UIViews. When the user taps any video view, the switchVideoViews() method is triggered to swap the positions of the large screen and the small screen (that is, the small window mentioned above).

[0080] Step 4, use the switchVideoViews() method to determine which view is the large screen through a Bool value isLargeVideoOnTop. Each time you click, switch this Boolean value and add a smooth animation for the switch through UIView.animate().

[0081] Step 5: When switching between large and small screens, add a 0.3 second animation to make the switching effect smoother. Use the UIView.animate() method to update the layout constraints in the animation block, and call self.view.layoutIfNeeded() to force an immediate layout update.

[0082] By manually managing NSLayoutConstraint, the position and size of the video view can be flexibly defined. For example, the click-to-switch function can be replaced with a drag gesture, and the user can switch by dragging the small screen to the large screen position.

[0083] You can also define resizing so that users can dynamically resize the small screen using a pinch gesture.

[0084] Or you can define the adjustment position, which can be expanded to multiple views to facilitate switching between multiple videos.

[0085] In view of the defect that traditional multi-camera monitoring can only display a single monitoring screen in full screen and cannot view multiple screens at the same time on the same screen, this application uses the picture-in-picture and floating window functions to allow users to view the main screen and the secondary screens of other cameras at the same time, breaking through the limitation of single-screen monitoring. In addition, the size and position of the floating window can be dynamically adjusted to adapt to different usage scenarios. The display position and size of the secondary monitoring screen can be adjusted at any time without conflicting with the main task. This flexible operation method is suitable for real-time monitoring needs in multi-tasking scenarios and significantly improves the user experience.

[0086] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0087] Figure 4 is a structural block diagram of a display device for multiple display screens according to an embodiment of the present application; Figure 4 As shown, including:

[0088] The receiving module 42 is used to parse the received video stream data to obtain the original video stream data collected by different image acquisition devices;

[0089] The preloading module 44 is used to decode the original video stream data to obtain video frame data, and obtain preloaded video frame data obtained after preloading the video frame data;

[0090] A determination module 46, configured to determine, from the preloaded video frame data, first video frame data corresponding to a first operation on a target object, and to determine, from the preloaded video frame data, second video frame data corresponding to a second operation on the target object, wherein the first operation represents selection of a first display page for displaying the first video frame data, and the second operation represents selection of a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in a display area of ​​a same terminal device, and a display format of the first display page and a display format of the second display page support mutual switching;

[0091] The display module 48 is used to display the first video frame data on the first display page, and to display the second video frame data on the second display page.

[0092] By means of the above-mentioned device, original video stream data from different image acquisition devices is received through a network transmission channel, wherein the network transmission channel represents a channel for transmitting the original video stream data between a terminal device and the different image acquisition devices; the original video stream data is decoded to obtain video frame data, and preloaded video frame data obtained after preloading the video frame data is obtained; first video frame data corresponding to a first operation of a target object is determined from the preloaded video frame data, and second video frame data corresponding to a second operation of the target object is determined from the preloaded video frame data, wherein the first operation represents selecting a first display page for displaying the first video frame data, and the second operation represents Select a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in the display area of ​​the same terminal device, and the display format of the first display page and the display format of the second display page support mutual switching; display the first video frame data on the first display page, and display the second video frame data on the second display page; by adaptively optimizing the videos collected by different cameras, the stability of video transmission is enhanced, and even in the case of large network fluctuations, the present application can still provide a stable video picture, and display multiple display pages in the terminal device according to user operations, thereby improving the display flexibility and efficiency of multiple display pages, and improving the adaptability of display pages and user operations. The above technical solution solves the technical problem of poor display effect of multiple display pictures, thereby improving the display effect and user experience of multiple display pictures.

[0093] In an exemplary embodiment, the preloading module is also used to: group the original video stream data according to the channel number of the network transmission channel to obtain multiple groups of video stream data to be decoded, wherein each group of video stream data to be decoded corresponds to an independent network transmission channel; and use video decoders to decode the multiple groups of video stream data to be decoded respectively to obtain decoded video frame data.

[0094] In an exemplary embodiment, the determination module is also used to determine the first video frame data corresponding to the first operation of the target object from the preloaded video frame data, and determine the second video frame data corresponding to the second operation of the target object from the preloaded video frame data, and then, upon receiving a switching operation of the display area by the target object, generate a first control instruction based on the switching operation; and use the first control instruction to switch the display format of the first display page and the display format of the second display page.

[0095] In an exemplary embodiment, the determination module is also used to: store the current frame data in the first video frame data to a first buffer, and preload the next frame data of the first video frame data to a second buffer; use the first control instruction to switch the display format of the first display page and the display format of the second display page, and at the same time read the next frame data of the first video frame data from the second buffer, and display the next frame data of the first video frame data on the first display page.

[0096] In an exemplary embodiment, the display module is also used to display the first video frame data on the first display page, and after displaying the second video frame data on the second display page, collect the first timestamp of the first video frame data and the second timestamp of the second video frame data according to a preset period; when it is determined that the timestamp difference between the first timestamp and the second timestamp is greater than the preset difference, if the playback time corresponding to the first timestamp is earlier than the playback time corresponding to the second timestamp, pause the display of the next frame data of the first video frame data on the first display page; or, if the playback time corresponding to the first timestamp is earlier than the playback time corresponding to the second timestamp, accelerate the display of the next frame data of the second video frame data on the second display page.

[0097] In an exemplary embodiment, the display module is also used to display the first video frame data on the first display page, and after displaying the second video frame data on the second display page: when it is monitored that the terminal device generates abnormal video stream data due to a playback failure, execute a reconnection instruction, re-establish a channel for transmitting the video stream data between the terminal device and the different image acquisition devices, and re-receive the video stream data; when it is determined that the re-received video stream data is displayed normally in the display area, determine the generation time of the abnormal video stream data, and determine the display time of the first frame data from the re-received video stream data; based on the comparison result of the display time and the generation time, update the re-received video stream data.

[0098] In an exemplary embodiment, the display module is also used to: when it is determined that the display moment is later than the generation moment, determine a first time period between the display moment and the generation moment, and obtain third video stream data belonging to the first time period from the original video stream data; and update the re-received video stream data based on the third video stream data.

[0099] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0100] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0101] S1, receiving raw video stream data from different image acquisition devices through a network transmission channel, wherein the network transmission channel refers to a channel for transmitting the raw video stream data between a terminal device and the different image acquisition devices;

[0102] S2, decoding the original video stream data to obtain video frame data, and obtaining preloaded video frame data obtained by preloading the video frame data;

[0103] S3, determining first video frame data corresponding to a first operation of a target object from the preloaded video frame data, and determining second video frame data corresponding to a second operation of the target object from the preloaded video frame data, wherein the first operation represents selection of a first display page for displaying the first video frame data, and the second operation represents selection of a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in a display area of ​​a same terminal device, and a display format of the first display page and a display format of the second display page support mutual switching;

[0104] S4, displaying the first video frame data on the first display page, and displaying the second video frame data on the second display page.

[0105] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0106] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0107] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0108] S1, receiving raw video stream data from different image acquisition devices through a network transmission channel, wherein the network transmission channel refers to a channel for transmitting the raw video stream data between a terminal device and the different image acquisition devices;

[0109] S2, decoding the original video stream data to obtain video frame data, and obtaining preloaded video frame data obtained by preloading the video frame data;

[0110] S3, determining first video frame data corresponding to a first operation of a target object from the preloaded video frame data, and determining second video frame data corresponding to a second operation of the target object from the preloaded video frame data, wherein the first operation represents selection of a first display page for displaying the first video frame data, and the second operation represents selection of a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in a display area of ​​a same terminal device, and a display format of the first display page and a display format of the second display page support mutual switching;

[0111] S4, displaying the first video frame data on the first display page, and displaying the second video frame data on the second display page.

[0112] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0113] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0114] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for displaying multiple display images, characterized in that: include: Receiving raw video stream data from different image acquisition devices through a network transmission channel, wherein the network transmission channel refers to a channel for transmitting the raw video stream data between a terminal device and the different image acquisition devices; Decoding the original video stream data to obtain video frame data, and acquiring preloaded video frame data obtained by preloading the video frame data; Determining first video frame data corresponding to a first operation of a target object from the preloaded video frame data, and determining second video frame data corresponding to a second operation of the target object from the preloaded video frame data, wherein the first operation represents selecting a first display page for displaying the first video frame data, and the second operation represents selecting a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in a display area of ​​the same terminal device, and a display format of the first display page and a display format of the second display page support mutual switching; The first video frame data is displayed on the first display page, and the second video frame data is displayed on the second display page.

2. The method for displaying multiple display images according to claim 1, characterized in that: Decoding the original video stream data to obtain video frame data includes: The original video stream data is grouped according to the channel number of the network transmission channel to obtain multiple groups of video stream data to be decoded, wherein each group of video stream data to be decoded corresponds to an independent network transmission channel; The plurality of groups of to-be-decoded video stream data are decoded using video decoders respectively to obtain decoded video frame data.

3. The method for displaying multiple display images according to claim 1, characterized in that: After determining the first video frame data corresponding to the first operation of the target object from the preloaded video frame data, and determining the second video frame data corresponding to the second operation of the target object from the preloaded video frame data, the method further includes: In case of receiving a switching operation of the target object on the display area, generating a first control instruction based on the switching operation; The first control instruction is used to switch the display format of the first display page and the display format of the second display page.

4. The method for displaying multiple display images according to claim 3, characterized in that: Using the first control instruction to switch the display format of the first display page and the display format of the second display page includes: storing current frame data in the first video frame data into a first buffer, and preloading next frame data of the first video frame data into a second buffer; The first control instruction is used to switch the display format of the first display page and the display format of the second display page, and at the same time, the next frame data of the first video frame data is read from the second buffer, and the next frame data of the first video frame data is displayed on the first display page.

5. The method for presenting multiple display images according to claim 1, characterized in that: After displaying the first video frame data on the first display page and displaying the second video frame data on the second display page, the method further includes: Collecting a first timestamp of the first video frame data and a second timestamp of the second video frame data according to a preset period; When it is determined that the timestamp difference between the first timestamp and the second timestamp is greater than a preset difference, if the playback time corresponding to the first timestamp is earlier than the playback time corresponding to the second timestamp, pausing display of the next frame of the first video frame data on the first display page; Alternatively, if the playback time corresponding to the first timestamp is earlier than the playback time corresponding to the second timestamp, the next frame data of the second video frame data is displayed faster on the second display page.

6. The method for presenting multiple display images according to claim 1, characterized in that: After displaying the first video frame data on the first display page and displaying the second video frame data on the second display page, the method further includes: In the case where it is detected that the terminal device generates abnormal video stream data due to a playback failure, executing a reconnection instruction, re-establishing a channel for transmitting the video stream data between the terminal device and the different image acquisition devices, and re-receiving the video stream data; In the case of determining that the re-received video stream data is normally displayed in the display area, determining the generation time of the abnormal video stream data, and determining the display time of the first frame data from the re-received video stream data; The re-received video stream data is updated based on the comparison result between the presentation time and the generation time.

7. The method for presenting multiple display images according to claim 6, characterized in that: The method of updating the re-received video stream data based on the comparison result between the display time and the generation time includes: In the case where it is determined that the presentation time is later than the generation time, determining a first time period between the presentation time and the generation time, and acquiring third video stream data belonging to the first time period from the original video stream data; The re-received video stream data is updated based on the third video stream data.

8. A display device with multiple display screens, characterized in that: include: A receiving module, used for receiving raw video stream data from different image acquisition devices through a network transmission channel, wherein the network transmission channel refers to a channel for transmitting the raw video stream data between a terminal device and the different image acquisition devices; A preloading module, used for decoding the original video stream data to obtain video frame data, and obtaining preloaded video frame data obtained after preloading the video frame data; A determination module, used to determine first video frame data corresponding to a first operation of a target object from the preloaded video frame data, and to determine second video frame data corresponding to a second operation of the target object from the preloaded video frame data, wherein the first operation represents selection of a first display page for displaying the first video frame data, and the second operation represents selection of a second display interface for displaying the second video frame data, wherein the first display page and the second display page are located in a display area of ​​a same terminal device, and a display format of the first display page and a display format of the second display page support mutual switching; A display module is used to display the first video frame data on the first display page and to display the second video frame data on the second display page.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images