Monitoring video display method, device and equipment and readable storage medium

Abstract

According to the monitoring video display method, device and equipment and the readable storage medium provided by the embodiment of the invention, the user roams in the digital twin world and issues the first frame selection instruction, and the electronic equipment responds to the first frame selection instruction and displays the monitoring video from the plurality of three-dimensional tags of the digital twin world under the first view angle. And determining the three-dimensional labels falling into the framing area of the first framing instruction to obtain a first label set. Then, the electronic equipment pulls a video stream of a camera corresponding to each three-dimensional tag in the first tag set to obtain multiple paths of video streams; and playing the multiple video streams on the user interface. By adopting the scheme, a plurality of monitoring pictures can be displayed through one-time frame selection, and the purposes of improving the display efficiency of the monitoring video and reducing misoperation are achieved. In addition, a plurality of cameras can be selected only through one-time multi-action, so that the operation steps of the user are reduced.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of digital twin technology, and in particular to a method, device, equipment and readable storage medium for displaying surveillance videos. Background Art

[0002] With the rapid development of Internet technology, the popularity of digital twins is increasing day by day. Based on digital twin technology, physical entities in the physical world can be mapped to the digital world. The virtual representation of a physical entity in the digital world is called a twin.

[0003] The digital world is actually a three-dimensional visual map. When there are a large number of surveillance cameras in the real physical world, these surveillance cameras are often marked in the three-dimensional digital world according to their locations in the real physical world. When you need to view a surveillance video, the user finds the corresponding camera in the three-dimensional digital world and clicks to watch. When you need to display multiple surveillance videos, click multiple times to call up multiple videos.

[0004] The above-mentioned display method of monitoring video is inefficient and prone to erroneous operation. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment and readable storage medium for displaying surveillance videos, which can display multiple surveillance screens by selecting a frame once, thereby achieving the purpose of improving the display efficiency of surveillance videos and reducing erroneous operations.

[0006] In a first aspect, an embodiment of the present application provides a method for displaying a surveillance video, comprising:

[0007] Displaying a digital twin world from a first-person perspective on a user interface, wherein the digital twin world is a virtual representation of a real physical world, and the digital twin world has a plurality of three-dimensional labels, wherein the three-dimensional labels represent cameras in the physical world;

[0008] In response to the first frame selection instruction, determining, from a plurality of three-dimensional labels of the digital twin world at the first perspective, three-dimensional labels that fall within a frame selection area of ​​the first frame selection instruction to obtain a first label set;

[0009] Pulling the video stream of the camera corresponding to each three-dimensional tag in the first tag set to obtain multiple video streams;

[0010] The multiple video streams are played on the user interface to display the multiple video streams.

[0011] In a second aspect, an embodiment of the present application provides a display device for monitoring video, including:

[0012] A display module, used to display a digital twin world from a first perspective on a user interface, wherein the digital twin world is a virtual representation of a real physical world, and the digital twin world has a plurality of three-dimensional labels, wherein the three-dimensional labels represent cameras in the physical world;

[0013] a processing module, configured to, in response to a first frame selection instruction, determine, from a plurality of three-dimensional labels of the digital twin world at the first perspective, three-dimensional labels that fall within a frame selection area of ​​the first frame selection instruction to obtain a first label set;

[0014] A pulling module, used to pull the video stream of the camera corresponding to each three-dimensional tag in the first tag set to obtain multiple video streams;

[0015] The playing module is used to play multiple video streams on the user interface to display the multiple video streams.

[0016] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method described in the first aspect or various possible implementation methods of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the method described in the first aspect or various possible implementation methods of the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product comprising a computing program, and when the computer program is executed by a processor, the method described in the first aspect or various possible implementation methods of the first aspect is implemented.

[0019] The display method, device, equipment and readable storage medium of the monitoring video provided by the embodiment of the present application, the user roams in the digital twin world and issues a first frame selection instruction, the electronic device responds to the first frame selection instruction, and determines the three-dimensional tags that fall into the frame selection area of ​​the first frame selection instruction from multiple three-dimensional tags in the digital twin world under the first perspective to obtain a first tag set. After that, the electronic device pulls the video stream of the camera corresponding to each three-dimensional tag in the first tag set to obtain multiple video streams; the multiple video streams are played on the user interface to display the multiple video streams. With this solution, when the user's frame selection operation in the interface of the three-dimensional digital twin world is recognized, the monitoring screens of multiple cameras in the frame selection area are displayed at the same time, and multiple monitoring screens can be displayed by one frame selection, so as to achieve the purpose of improving the display efficiency of the monitoring video and reducing misoperation. Moreover, multiple cameras can be selected with only one multi-action, which reduces the user's operation steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 It is a schematic diagram of a traditional two-dimensional park;

[0022] Figure 2 It is a schematic diagram of multiple video streams in a three-dimensional campus interface;

[0023] Figure 3 It is a schematic diagram of an implementation environment applicable to the display method of monitoring video provided in the embodiment of the present application;

[0024] Figure 4 is a flowchart of a method for displaying surveillance video provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of a frame selection area of ​​the monitoring video display method provided in an embodiment of the present application;

[0026] Figure 6 It is a schematic diagram of monitoring screen display of the monitoring video display method provided in the embodiment of the present application;

[0027] Figure 7 It is a schematic diagram of the layout of the monitoring screen in the monitoring video display method provided in the embodiment of the present application;

[0028] Figure 8 It is a process schematic diagram of a method for displaying surveillance video provided in an embodiment of the present application;

[0029] Fig. 9 is a schematic diagram of an outer bounding box in a method for displaying a surveillance video provided in an embodiment of the present application;

[0030] Fig.10 is a schematic diagram of a display device for monitoring video provided in an embodiment of the present application;

[0031] Fig.11 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] A park is an area of ​​unified planning and construction. The supporting facilities such as water supply, power supply, equipment, etc. in the park can meet the needs of a specific industry. Common parks include university parks, industrial parks, science and technology parks, etc. In order to facilitate the management of the park, smart parks are booming. Smart parks use technical means such as maps and the Internet of Things to achieve comprehensive perception, interconnection and intelligent management of people, things and objects in the park. For example, a park is equipped with a large number of cameras, and the icons of these cameras are displayed in the smart park. The position of each icon in the smart park is consistent with the actual position of the camera in the physical park. By clicking the icon on the smart park interface, the video stream of the corresponding camera can be displayed to view the surveillance video of the camera.

[0033] Traditional smart parks include two-dimensional parks and three-dimensional parks. Figure 1 This is a schematic diagram of a traditional two-dimensional park. Please refer to Figure 1 , the two-dimensional campus is similar to a two-dimensional plane map, on which a large number of camera icons are displayed, each icon representing a camera in the real physical world, as shown in the circle in the figure. It has been verified that the two-dimensional campus cannot restore the real physical world, resulting in the user being unable to combine the camera's monitoring screen with the actual site of occurrence, and can only imagine the site of occurrence based on the monitoring screen. Moreover, a point in the two-dimensional campus does not represent a point in the real physical world, but a series of locations in the real physical world. For example, there is a 6-story teaching building in the real physical world, and cameras are installed at the same position on each floor. Then, there are 6 camera icons at the same point in the two-dimensional campus, representing cameras on different floors. These 6 icons are very close or even overlapped, making it difficult for users to select the icon of the camera they want to view from the 6 icons. Moreover, the number of cameras in a campus is thousands or even more, which makes it difficult for users to select the camera icons they want to view from the 6 icons. In addition, the number of cameras in a campus is thousands or even more, which makes it difficult for users to select the camera icons that they want to display the monitoring screen on the two-dimensional campus interface, which makes it difficult for users to select the camera they want to display the monitoring screen, and thus makes it difficult to view the monitoring video.

[0034] Compared with the two-dimensional campus, the three-dimensional campus can restore the real physical world through technologies such as digital twins. Users find the icon of the camera they want to display the monitoring screen on the three-dimensional campus interface and click it to view the camera's monitoring screen. For example, if the user clicks the icon of the camera on the second floor of the teaching building, the monitoring screen of the camera on the second floor will be displayed on the three-dimensional campus interface.

[0035] However, whether it is a 2D campus or a 3D campus, the interactive mode of viewing surveillance is point-based. In other words, one click only supports opening one camera for display. If the user wants to display and view multiple cameras at the same time, multiple clicks are required. Figure 2 This is a schematic diagram of multiple video streams in the 3D campus interface. Please refer to Figure 2 , the user clicks on the icons of the four cameras in turn, and the interface displays four video streams. Among the four video streams, the two video streams that are closer are blocked, resulting in the inability to display multiple images at the same time. In addition, the video streams of the cameras that are farther away are scattered, which also makes it difficult to display all video streams at the same time.

[0036] Obviously, the above-mentioned monitoring video display method requires the user to click on each camera one by one, which is very time-consuming when there are many cameras. Moreover, the operation steps are cumbersome, and multiple click operations may cause user fatigue and wrong operations. When it is necessary to pay attention to multiple monitoring screens at the same time, the one-by-one display method is difficult to achieve synchronous monitoring. In addition, the switching cost is high. This is because after the user displays the monitoring screen of a camera under one viewing angle, if the user needs to display the monitoring screen of other cameras under another viewing angle, he must exit the current viewing angle and then make a selection, which increases the switching cost.

[0037] Although some parks have monitoring rooms with a large number of display screens, and all camera surveillance images are displayed on the display screens, it is difficult for security guards or staff to clearly remember or remember the corresponding cameras in each area of ​​the park. When a certain area needs to be paid attention to, it is difficult to find the surveillance images of that area from all the surveillance images.

[0038] Based on this, the embodiments of the present application provide a method, device, equipment and readable storage medium for displaying surveillance videos. When a user's frame selection operation in the interface of a three-dimensional digital twin world is recognized, the surveillance images of multiple cameras in the frame selection area are displayed at the same time. Multiple surveillance images can be displayed through a single frame selection, thereby achieving the purpose of improving the display efficiency of surveillance videos and reducing misoperations.

[0039] Figure 3 This is a schematic diagram of an implementation environment applicable to the monitoring video display method provided in the embodiment of the present application. Figure 3 , the implementation environment includes a terminal device 31 and a server 32. A network connection is established between the terminal device 31 and the server 32.

[0040] The terminal device 31 is, for example, a smart interactive tablet, a laptop computer, a desktop computer, a mobile phone, a tablet computer, etc., which is not limited in the embodiments of the present application.

[0041] The server 32 may be an independent physical server, a server cluster composed of multiple servers, or a cloud server with cloud computing capabilities, etc., which is not limited by the embodiments of the present application. The server 32 stores data related to the digital twin world and video streams pulled by each surveillance camera. The server for storing video streams and the server for storing data related to the digital twin world may be the same server or different servers, which is not limited by the embodiments of the present application.

[0042] In the embodiment of the present application, various technologies such as drone oblique photography and laser point cloud panoramic cameras are used in advance to complete rapid real-scene modeling of the real physical world, thereby creating a digital twin world corresponding to the physical world. The digital twin world can be understood as: a world created in the virtual world that is a 1:1 replica of the real physical world, used to restore the real physical world. For each physical entity in the physical world, there is a corresponding twin in the digital twin world. The digital twin world is actually a three-dimensional map that can be enlarged, reduced, and rotated. When the user roams in the digital twin world, he can overlook the entire digital twin world, view the digital twin world from different perspectives, or take a detailed look at a twin in the digital twin world.

[0043] When the real physical world is a campus, the digital twin world is a smart campus; when the real physical world is a factory, the digital twin world is a smart project; when the real physical world is a museum, the digital twin world is a smart museum... The embodiments of the present application do not limit the type of the real physical world corresponding to the digital twin world.

[0044] The terminal device 31 is installed with web applications such as a browser, and the user can enter the URL of the digital twin world, etc., to open the web site of the digital twin world and roam in the digital twin world. During the roaming process, the user switches the perspective at the current zoom level by controlling the mouse, or enters a different zoom level to switch the perspective. For example, teaching building A and teaching building B are at the same zoom level, and the current interface shows teaching building A but does not show teaching building B. The user controls the mouse to call out teaching building B to switch the perspective; for another example, the current interface shows teaching building A, and the user enters teaching building A by roaming, that is, changes the zoom level to switch the perspective.

[0045] When the user roams in the digital twin world, he selects multiple cameras by means of box selection. After the terminal device 31 detects the box selection operation, it determines the multiple cameras falling into the box selection area, and displays the monitoring images of the multiple cameras in the box selection area at the same time, so that multiple monitoring images can be displayed through a single box selection, thereby achieving the purpose of improving the display efficiency of the monitoring video and reducing misoperation. Afterwards, the user can rearrange the multiple monitoring images to watch multiple video streams at the same time.

[0046] In the embodiment of the present application, the monitoring images of multiple cameras can be displayed simultaneously, which can be real-time monitoring images of multiple cameras, or historical monitoring images of multiple cameras. For example, three cameras are located in the playground, the cafeteria and the campus intersection, and the monitoring images of the three cameras at 10 o'clock last night can be displayed simultaneously.

[0047] It should be noted that although Figure 3 A server 32 is provided. However, the embodiments of the present application are not limited thereto. For example, when the terminal device 31 is a device with strong data processing capabilities, the data related to the digital twin world can also be saved on the terminal device 31, so that even if the terminal device 31 is not connected to the Internet, the terminal device 31 can simultaneously display the monitoring images of multiple cameras during the user's panoramic roaming at any time and anywhere.

[0048] It should be understood that Figure 3 The number of terminal devices 31 and servers 32 in the figure is only for illustration. In actual implementation, any number of terminal devices 31 and servers 32 can be deployed according to actual needs.

[0049] Below, based on Figure 3 The implementation environment shown is used to describe in detail the display method of the surveillance video described in the embodiment of the present application. For example, please refer to Figure 4 , Figure 4 1 is a flowchart of a method for displaying surveillance video provided by an embodiment of the present application. The execution subject of the embodiment of the present application is an electronic device, such as the above-mentioned terminal device or server, and the embodiment includes:

[0050] 401. Display a digital twin world from a first perspective on a user interface, where the digital twin world is a virtual representation of a real physical world. The digital twin world has a plurality of three-dimensional labels, and the three-dimensional labels represent cameras in the physical world.

[0051] Exemplarily, a user opens the digital twin world through a web application and roams in the digital twin world. During the roaming process, it is equivalent to having a digital person representing the user walking in the digital twin world. The digital person can be displayed in the digital twin world or not. The digital person is like a camera that can simulate the user's perspective. During the roaming process, the user can zoom in, zoom out, and rotate the digital twin world by controlling the mouse to switch the perspective. The user can perform a frame selection operation at any perspective. The user's perspective when the frame is selected for the first time is referred to as the first perspective.

[0052] The digital twin world has a large number of twins and three-dimensional labels that annotate the twins. Twins are, for example, teaching buildings, playgrounds, canteens, gates, cameras, etc. The three-dimensional labels in the embodiments of the present application refer to the three-dimensional labels of cameras in the digital twin world.

[0053] 402. In response to a first selection instruction, determine, from a plurality of three-dimensional labels of the digital twin world from the first perspective, three-dimensional labels that fall within a selection area of ​​the first selection instruction to obtain a first label set.

[0054] When roaming to a certain location, if the user wants to view the surveillance, he / she issues a first frame selection instruction by dragging the mouse, sliding the finger, etc. After the electronic device recognizes the first frame selection instruction, it determines the three-dimensional labels falling into the frame selection area from multiple three-dimensional labels of the digital twin world under the current first perspective to obtain the first label set.

[0055] Figure 5 is a schematic diagram of a frame selection area of ​​the monitoring video display method provided in the embodiment of the present application, which shows a part of the digital twin world from the first perspective. Figure 5 After the user drags out a selection area with a mouse, the electronic device determines the three-dimensional tags falling into the selection area from multiple three-dimensional tags in the digital twin world from the first perspective according to the position and size of the selection area, the position and size of each three-dimensional tag, such as dormitory monitoring 3, dormitory monitoring 6 and canteen monitoring 1 shown in the figure. The identifiers (ID) of the three three-dimensional tags form a first tag set.

[0056] 403 . Pull the video stream of the camera corresponding to each three-dimensional tag in the first tag set to obtain multiple video streams.

[0057] Exemplarily, the electronic device opens multiple video pop-up windows through a web application such as a browser. When the pop-up windows are initialized, they query the video service one by one through Hypertext Transfer Protocol (HTTP) requests to obtain the video stream address corresponding to each ID, and pull the video stream according to each video stream address.

[0058] 404. Play the multiple video streams on the user interface to display the multiple video streams.

[0059] Exemplarily, the web application on the electronic device creates a player for each video stream, and the video stream address and the player correspond one-to-one, that is, a player is created for the camera corresponding to each three-dimensional tag, and the video stream is played and displayed through the player.

[0060] Figure 6 Schematic diagram of monitoring screen display of the monitoring video display method provided in the embodiment of the present application. Figure 6 , three players are displayed from top to bottom on the right, which are used to play the surveillance images from dormitory monitoring 3, dormitory monitoring 6 and canteen monitoring 1 respectively, so that users can view the surveillance videos of dormitory monitoring 3, dormitory monitoring 6 and canteen monitoring 1 at the same time.

[0061] Please refer to Figure 6 The user interface also displays the number of cameras selected. All players are located in a pop-up window, as shown in the thick black rectangle in the figure. The pop-up window can be moved to any position in the user interface.

[0062] It should be noted that since the user only selected three cameras, Figure 6 The surveillance images of the three cameras are displayed simultaneously. When the number of cameras selected is relatively large, such as 10, 8, 15 or even more, the electronic device automatically calculates the width and height of the player and arranges each player. When the pop-up window cannot accommodate each player, the preset number of players is displayed and the remaining players are hidden. For example, when multiple players are arranged in a row, Figure 6 The pop-up window shown shows a maximum of 4 players. When the number of cameras selected is greater than 4, 4 players are displayed in the pop-up window, that is, the monitoring screens of 4 cameras are displayed, and the monitoring screens of the remaining players are hidden. When the user clicks the hide icon (not shown in the figure), the pop-up window switches to other players and displays them.

[0063] In the display method of monitoring video provided by the embodiment of the present application, the user roams in the digital twin world and issues a first frame selection instruction. The electronic device responds to the first frame selection instruction and determines the three-dimensional tags that fall into the frame selection area of ​​the first frame selection instruction from multiple three-dimensional tags in the digital twin world under the first perspective to obtain a first tag set. After that, the electronic device pulls the video stream of the camera corresponding to each three-dimensional tag in the first tag set to obtain multiple video streams; the multiple video streams are played on the user interface for the user to display the multiple video streams. With this solution, when the user's frame selection operation in the interface of the three-dimensional digital twin world is recognized, the monitoring screens of multiple cameras in the frame selection area are displayed at the same time, so that the user can display multiple monitoring screens by one frame selection, thereby improving the display efficiency of the monitoring video and reducing the purpose of misoperation. Moreover, multiple cameras can be selected with only one multi-action, which reduces the user's operation steps.

[0064] Optionally, in the above embodiment, after the electronic device plays the multiple video streams on the user interface, it also responds to the second frame selection instruction and determines the three-dimensional tags that fall within the frame selection area of ​​the second frame selection instruction to obtain a second tag set. Afterwards, the electronic device pulls the video stream of the camera corresponding to each three-dimensional tag in the union of the first tag set and the second tag set to obtain multiple video streams; and plays the multiple video streams on the user interface.

[0065] For example, the embodiment of the present application supports multiple selections by the user as needed, and two adjacent selections are not in an overlapping relationship. For example, the first selection obtains the first tag set, and the second selection obtains the second tag set. The second tag set will not overwrite the first tag set, but will be based on the existing tag set, through comparison, deduplication, and merging to obtain the union of the first tag set and the second tag set, and finally pull the video stream of the camera corresponding to each three-dimensional tag in the union to obtain multiple video streams, and play the multiple video streams on the user interface.

[0066] This solution supports users to select multiple cameras as needed, making it easier for users to select multiple cameras that need to be monitored simultaneously without requiring users to select cameras one by one multiple times, thereby improving the display efficiency of surveillance videos.

[0067] When the user makes multiple selections, in one method, the second selection and the first selection are both selections of the three-dimensional labels on the digital twin interface from the first perspective.

[0068] In this manner, the electronic device responds to the second selection instruction and determines, from the multiple three-dimensional tags on the digital twin interface under the first viewing angle, the three-dimensional tags that fall within the selection area of ​​the second selection instruction to obtain the second tag set. Then, a union is determined based on the first tag set and the second tag set.

[0069] For example, please refer to Figure 5 , the first label set includes dormitory monitoring 3, dormitory monitoring 6, and canteen monitoring 1. The second label set obtained by the second selection includes dormitory monitoring 3, dormitory monitoring 6, canteen monitoring 1, dormitory monitoring 1, and activity center monitoring. The electronic device obtains a union by comparison, deduplication, and merging, and the union includes dormitory monitoring 3, dormitory monitoring 6, canteen monitoring 1, dormitory monitoring 1, and activity center monitoring.

[0070] For example, please refer to Figure 5 , the first label set includes dormitory monitoring 3, dormitory monitoring 6, and canteen monitoring 1. The second label set obtained by the second selection includes dormitory monitoring 1 and activity center monitoring, and the union includes dormitory monitoring 3, dormitory monitoring 6, canteen monitoring 1, dormitory monitoring 1, and activity center monitoring.

[0071] For example, please refer to Figure 5 , the first label set includes dormitory monitoring 3, dormitory monitoring 6 and canteen monitoring 1. The second label set obtained by the second selection includes dormitory monitoring 3, dormitory monitoring 6 and dormitory monitoring 7, and the union includes dormitory monitoring 3, dormitory monitoring 6, canteen monitoring 1 and dormitory monitoring 7.

[0072] With this solution, users can select multiple times from the same viewing angle to select the surveillance that needs to be displayed, avoiding the situation where the result of a single selection does not meet expectations. This solution is highly flexible and achieves the purpose of improving the display efficiency and flexibility of surveillance videos.

[0073] In another method, the first selection is to select the three-dimensional label on the digital twin interface in the first perspective, and the second selection is to select the three-dimensional label on the digital twin interface in the second perspective.

[0074] For example, please refer to Figure 5 , the digital twin interface in the first perspective is the northern area of ​​the campus. After the electronic device obtains the first label set according to the first selection instruction, the user roams by controlling the mouse, etc., and roams to the southern area of ​​the campus, that is, the digital twin interface in the second perspective is the southern area of ​​the campus, and the cameras in the southern area include library monitoring 4, computer building monitoring 5, school auditorium monitoring 2, etc. The user selects the three-dimensional labels on the digital twin interface in the second perspective, and the electronic device recognizes that the second label set includes computer building monitoring 5 and school auditorium monitoring 2, then the union includes dormitory monitoring 3, dormitory monitoring 6, canteen monitoring 1, computer building monitoring 5 and school auditorium monitoring 2.

[0075] For example, please refer to Figure 5After the electronic device obtains the first tag set according to the first selection instruction, the user roams inside the canteen by controlling the mouse. The canteen has three floors, with cameras a, b, and c set from bottom to top. The user currently roams to the second floor, that is, the digital twin interface under the second perspective is the second floor of the canteen. The user selects camera b, and the union includes dormitory monitoring 3, dormitory monitoring 6, canteen monitoring 1, and camera b.

[0076] By adopting this solution, the user can select the monitoring that needs to be displayed by making multiple selections at different perspectives, so as to simultaneously display the monitoring images of cameras at different levels of the digital twin interface, thereby improving the display efficiency and flexibility of the monitoring video.

[0077] Optionally, in the above embodiment, after playing the multiple video streams on the user interface, the electronic device further responds to the layout instruction and layouts the multiple players according to the layout method indicated by the layout instruction. Afterwards, the electronic device displays the multiple players on the user interface according to the layout method, and the players in the multiple players correspond to the video streams in the multiple video streams one by one, and each player is used to play the corresponding video stream.

[0078] Please refer to Figure 6 The default layout mode is that multiple players are displayed in one column and a preset number of players are displayed and the remaining players are hidden. Different players are used to play different video streams. Users can set the default layout mode. The layout modes include but are not limited to: multiple players are displayed in one column and a preset number of players are displayed and the remaining players are hidden, the multiple players are displayed in multiple rows and columns and the number of rows and columns is at least two, and one of the multiple players is displayed in full screen and the remaining players are hidden.

[0079] Among the above layout modes, the layout mode of one column layout and the layout mode of multiple rows and columns layout are convenient for users to watch multiple video streams at the same time, that is, to watch the monitoring screens of multiple cameras at the same time. Full screen display is convenient for users to carefully display a certain video stream among multiple video streams. The layout mode of one column layout and the layout mode of multiple rows and columns layout are also called split screen display.

[0080] By adopting this solution, users can layout the monitoring screens of the selected cameras according to their needs, so that multiple monitoring screens are displayed reasonably with high flexibility, thereby achieving the purpose of improving the display efficiency of monitoring videos.

[0081] Figure 7 This is a schematic diagram of the layout of the monitoring screen in the monitoring video display method provided in the embodiment of the present application. Figure 7, 7 buttons are displayed at the bottom of the pop-up window, namely button 71, button 72, button 73, button 74, button 75, button 76 and button 77. Button 71 represents one-column layout, button 72 represents two-row and two-column layout, button 73 represents three-row and three-column layout, button 74 represents displaying the previous page, button 75 represents displaying the next page, and button 77 represents returning to the default layout mode and position.

[0082] After the user selects 11 cameras and clicks button 72, the pop-up window displays the players in a 2-row 2-column layout, that is, the surveillance images of 4 cameras are displayed simultaneously. When the user clicks button 76, the pop-up window switches to the surveillance images of another 4 cameras. Under different layout methods, the electronic device calculates the length and width of each player and displays each player according to the layout method. Different players are used to broadcast the surveillance images of different cameras.

[0083] By adopting this solution, the electronic device typeset and displays multiple players according to the typesetting instructions. The operation is simple and the purpose of flexibly typeset the players is achieved.

[0084] Figure 8 This is a schematic diagram of the process of displaying the monitoring video provided by the embodiment of the present application. Figure 8 , this embodiment includes the following steps:

[0085] 801. Enable frame selection mode.

[0086] Users open the digital twin world through web applications, and cameras in the physical world are represented by three-dimensional labels in the three-dimensional digital twin world, such as Figure 5 to Figure 7 As shown. The electronic device projects each 3D tag to the screen coordinate system to obtain multiple projection images, and constructs an outer bounding box for each of the multiple projection images to obtain multiple outer bounding boxes. That is, each 3D tag has its own outer bounding box.

[0087] In the process of creating the outer bounding box, the electronic device dynamically constructs the outer bounding box based on the coordinate sequence of the 3D tag, supporting polygon construction of any complexity. When the outer bounding box is a rectangle, the electronic device records the coordinates of the upper left corner vertex and the lower right corner vertex of the rectangle; when the outer bounding box is a polygon, the electronic device records the coordinates of all vertices.

[0088] Fig. 9 Schematic diagram of the outer bounding box in the monitoring video display method provided in the embodiment of the present application. Fig. 9 , the rectangular box is the outer bounding box. It should be noted that this is only for showing the appearance and position of the rectangular outer bounding box. The user interface will not display the outer bounding box during the user's actual roaming process, that is, the outer bounding box is invisible to the user.

[0089] After the electronic device completes the rendering of the digital twin interface, the creation of the outer bounding box of the camera three-dimensional label, etc., the interface of the digital twin world is displayed on the user interface, and the user roams in the digital twin world. The interface of the digital twin world has a selection button, and the mouse clicks the selection button to enter the selection mode. After entering the selection mode, the electronic device responds to the first selection instruction, and according to each outer bounding box in the multiple outer bounding boxes and the selection area of ​​the first selection instruction, determines the three-dimensional label falling into the selection area of ​​the first selection instruction to obtain the first label set.

[0090] With this solution, the electronic device projects each 3D label onto the screen coordinate system and creates an outer bounding box. During the selection process, it determines whether the 3D label falls into the selection area based on the outer bounding box and the selection area. This solution is fast and accurate.

[0091] 802. Select the camera whose surveillance image needs to be displayed.

[0092] The user drags the mouse to adjust the size and position of the selection area, thereby issuing a first selection instruction to select the camera that needs to display the monitoring screen.

[0093] 803. Complete the box selection and obtain the first label set.

[0094] Exemplarily, the user adjusts the selection area by dragging, and the electronic device uses the SAT algorithm to retrieve an outer bounding box that intersects with the selection area. If an outer bounding box intersects with the selection area, the three-dimensional label corresponding to the outer bounding box is used as the three-dimensional label that falls into the selection area, that is, as the three-dimensional label in the first label set.

[0095] In the process of the electronic device using the SAT algorithm to determine the first label set, multiple separation axes are determined according to the selection area of ​​the first selection instruction, and the normal vector of each edge of the selection area can be used as the separation axis. For each outer bounding box in the multiple outer bounding boxes, the projections of the outer bounding box and the selection area on each separation axis are determined. Afterwards, the electronic device determines whether there are intersecting separation axes among the multiple separation axes. When there is at least one intersecting separation axis among the multiple separation axes, it is determined that the label corresponding to the outer bounding box falls into the selection area of ​​the first selection instruction; when there is no intersecting separation axis among the multiple separation axes, the electronic device determines that the label corresponding to the outer bounding box does not fall into the selection area. Among them, the intersecting separation axis means that the projections of the outer bounding box and the selection area of ​​the first selection instruction on the intersecting separation axis overlap.

[0096] In the above method for determining the first label set, after the electronic device obtains the separation axis, for each separation axis, it determines the projection of the frame selection area and the outer bounding box on the axis. If the projections of the frame selection area and the outer bounding box on each separation axis do not overlap, it means that the frame selection area and the outer bounding box do not intersect; if the projections of the frame selection area and the outer bounding box on at least one separation axis overlap, it means that the frame selection area and the outer bounding box intersect, and this separation axis is an intersecting separation axis, and the three-dimensional label corresponding to the outer bounding box falls into the frame selection area.

[0097] With this solution, the electronic device uses the separating axis theorem to perform intersection detection, thereby achieving the purpose of quickly and accurately determining the camera to be selected.

[0098] 804. When the next round of selection operation is required, return to step 801 to restart the selection mode.

[0099] The embodiment of the present application supports users to select boxes multiple times as needed. The label set obtained each time the box is selected is not an overlapping relationship. Instead, a new ID set, i.e., a union, is obtained by comparing, deduplicating, and merging the selected label IDs.

[0100] 805. The electronic device sends the IDs of the three-dimensional tags to the web application.

[0101] 806. After receiving the ID of each three-dimensional tag, the web application of the electronic device opens a pop-up window.

[0102] 807. The electronic device queries the address of the video stream corresponding to each ID.

[0103] Exemplarily, when the pop-up window is initialized, the electronic device queries the video service for the address of the video stream corresponding to each ID through HTTP requests one by one. This step includes the following sub-steps:

[0104] 8071. The electronic device sends an ID to the video service.

[0105] 8072. The electronic device receives the address of the video stream corresponding to the ID from the video service.

[0106] Exemplarily, after receiving the ID, the video server determines the address of the video stream corresponding to the ID and returns it to the electronic device. The address of the video stream is, for example, the uniform resource locator (URL) of the stream.

[0107] 808. The electronic device pulls the video stream from the video service according to the address of the video stream.

[0108] 809. The electronic device creates a player for each video stream.

[0109] 810. The electronic device calculates the width and height of the player and arranges the layout.

[0110] 811. The electronic device monitors the user's typesetting instructions.

[0111] 812. When the layout instructions such as full screen switching and split screen mode are monitored, the width and height of the player are recalculated and the layout is arranged.

[0112] 813. Display the monitoring screen of the corresponding camera through the player.

[0113] Exemplarily, multiple players are displayed on the user interface according to the layout method, and each player is used to play a corresponding video stream.

[0114] 814. Close the pop-up window.

[0115] 815. Destroy the player and stop streaming.

[0116] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0117] Fig.10 1 is a schematic diagram of a monitoring video display device provided in an embodiment of the present application. The monitoring video display device 1000 comprises: a display module 101 , a processing module 102 , a pulling module 103 and a playing module 104 .

[0118] A display module 101 is used to display a digital twin world from a first perspective on a user interface, where the digital twin world is a virtual representation of a real physical world, and the digital twin world has a plurality of three-dimensional labels, where the three-dimensional labels represent cameras in the physical world;

[0119] The processing module 102 is configured to, in response to the first frame selection instruction, determine, from a plurality of three-dimensional labels of the digital twin world at the first perspective, three-dimensional labels that fall within a frame selection area of ​​the first frame selection instruction to obtain a first label set;

[0120] A pulling module 103 is used to pull the video stream of the camera corresponding to each three-dimensional tag in the first tag set to obtain multiple video streams;

[0121] The playing module 104 is used to play the multiple video streams on the user interface to display the multiple video streams.

[0122] In a feasible implementation, after the playing module 104 plays the multiple video streams on the user interface, the processing module 102 is further configured to respond to a second frame selection instruction and determine the three-dimensional tags falling within the frame selection area of ​​the second frame selection instruction to obtain a second tag set;

[0123] The pulling module 103 is further used to pull the video stream of the camera corresponding to each three-dimensional tag in the union of the first tag set and the second tag set to obtain multiple video streams;

[0124] The playing module 104 is used to play multiple video streams on the user interface.

[0125] In a feasible implementation, the processing module 102 is used to respond to the second selection instruction and determine, from multiple three-dimensional labels on the digital twin interface under the first perspective, the three-dimensional labels that fall within the selection area of ​​the second selection instruction to obtain the second label set.

[0126] In a feasible implementation, the processing module 102 is used to switch the user interface to a digital twin world under a second perspective in response to a roaming instruction; and in response to a second frame selection instruction, determine, from multiple three-dimensional labels on the digital twin world under the second perspective, the three-dimensional labels that fall within the frame selection area of ​​the second frame selection instruction to obtain the second label set.

[0127] In a feasible implementation, the processing module 102, after the playing module 104 plays the multiple video streams on the user interface, is further used to respond to the layout instruction and layout the multiple players according to the layout method indicated by the layout instruction, and the players in the multiple players correspond to the video streams in the multiple video streams one by one;

[0128] The display module 101 is used to display the multiple players on the user interface according to the layout method, and each player is used to play a corresponding video stream.

[0129] In a feasible implementation, the layout method includes at least one of the following layout methods: the multiple players are displayed in a column and a preset number of players are displayed and the remaining players are hidden, the multiple players are displayed in multiple rows and columns and the number of rows and columns is at least two, and one of the multiple players is displayed in full screen and the remaining players are hidden.

[0130] In a feasible implementation, the processing module 102 is used to project each of the multiple three-dimensional labels to a screen coordinate system to obtain multiple projection images; construct an outer bounding box for each of the multiple projection images to obtain multiple outer bounding boxes; and in response to the first frame selection instruction, determine the three-dimensional labels that fall within the frame selection area of ​​the first frame selection instruction according to each outer bounding box of the multiple outer bounding boxes and the frame selection area of ​​the first frame selection instruction to obtain a first label set.

[0131] In a feasible implementation, the processing module 102 responds to the first frame selection instruction, and determines the three-dimensional label falling within the frame selection area of ​​the first frame selection instruction according to each outer bounding box in the multiple outer bounding boxes and the frame selection area of ​​the first frame selection instruction to obtain a first label set. It is used to respond to the first frame selection instruction and determine multiple separation axes according to the frame selection area of ​​the first frame selection instruction; for each outer bounding box in the multiple outer bounding boxes, determine the projections of the outer bounding box and the frame selection area of ​​the first frame selection instruction on each separation axis; when there is at least one intersecting separation axis among the multiple separation axes, determine that the label corresponding to the outer bounding box falls within the frame selection area of ​​the first frame selection instruction, and the projections of the outer bounding box and the frame selection area of ​​the first frame selection instruction on the intersecting separation axis overlap.

[0132] The display device for surveillance video provided in the embodiment of the present application can execute the actions of the electronic device in the above-mentioned embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0133] Fig.11 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. Fig.11 The electronic device 1100 described in the embodiment of the present application includes: at least one processor 111, at least one communication bus 112, a user interface 113, at least one network interface 114 and a memory 115.

[0134] The communication bus 112 is used to realize the connection and communication between these components.

[0135] The user interface 113 may include a display screen (Display) and a camera (Camera). The optional user interface 113 may also include a standard wired interface and a wireless interface. The display screen is used to display an editing interface, a roaming interface, and the like.

[0136] The network interface 114 may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface).

[0137] Among them, the processor 111 may include one or more processing cores. The processor 111 uses various interfaces and lines to connect various parts within the entire electronic device 1100, and executes various functions and processes data of the electronic device 1100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 115, and calling data stored in the memory 115. Optionally, the processor 111 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 111 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processor (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is used for image processing; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 111, and it can be implemented separately through a chip.

[0138] Among them, the memory 115 may include a random access memory (Random Access Memory, RAM) and may also include a read-only memory (Read-Only Memory). Optionally, the memory 115 includes a non-transitory computer-readable storage medium. The memory 115 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 115 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 115 may also be optionally at least one storage device located away from the aforementioned processor 111. As Fig.11 As shown, the memory 115 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an operating application of the electronic device.

[0139] An embodiment of the present application also provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the display method of the surveillance video implemented by the electronic device as above.

[0140] An embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it implements the monitoring video display method implemented by the above electronic device.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0142] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0144] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0145] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0146] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0147] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0148] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0149] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for displaying a surveillance video, characterized in that: Applied to electronic equipment, the method comprises: Displaying a digital twin world from a first-person perspective on a user interface, wherein the digital twin world is a virtual representation of a real physical world, and the digital twin world has a plurality of three-dimensional labels, wherein the three-dimensional labels represent cameras in the physical world; In response to the first frame selection instruction, determining, from a plurality of three-dimensional labels of the digital twin world at the first perspective, three-dimensional labels that fall within a frame selection area of ​​the first frame selection instruction to obtain a first label set; Pulling the video stream of the camera corresponding to each three-dimensional tag in the first tag set to obtain multiple video streams; The multiple video streams are played on the user interface to display the multiple video streams.

2. The method according to claim 1, characterized in that After playing the multiple video streams on the user interface, the method further includes: In response to a second frame selection instruction, determining three-dimensional labels falling within a frame selection area of ​​the second frame selection instruction to obtain a second label set; Pulling the video stream of the camera corresponding to each three-dimensional tag in the union of the first tag set and the second tag set to obtain multiple video streams; Play multiple video streams on the user interface.

3. The method according to claim 2, characterized in that In response to the second frame selection instruction, determining the three-dimensional labels falling within the frame selection area of ​​the second frame selection instruction to obtain a second label set includes: In response to the second frame selection instruction, the three-dimensional labels falling within the frame selection area of ​​the second frame selection instruction are determined from the multiple three-dimensional labels on the digital twin interface under the first perspective to obtain the second label set.

4. The method according to claim 2, characterized in that: In response to the second frame selection instruction, determining the three-dimensional labels falling within the frame selection area of ​​the second frame selection instruction to obtain a second label set includes: In response to the roaming instruction, the user interface is switched to the digital twin world in the second perspective; In response to the second frame selection instruction, the three-dimensional labels falling within the frame selection area of ​​the second frame selection instruction are determined from the multiple three-dimensional labels on the digital twin world from the second perspective to obtain the second label set.

5. The method according to any one of claims 1 to 4, characterized in that: After playing the multiple video streams on the user interface, the method further includes: In response to the layout instruction, layout a plurality of players according to the layout mode indicated by the layout instruction, wherein the players in the plurality of players correspond to the video streams in the plurality of video streams in a one-to-one manner; The multiple players are displayed on the user interface according to the layout method, and each player is used to play a corresponding video stream.

6. The method according to claim 5, characterized in that The layout method includes at least one of the following layout methods: the multiple players are displayed in one column and a preset number of players are displayed and the remaining players are hidden; the multiple players are displayed in multiple rows and columns and the number of rows and columns is at least two; one of the multiple players is displayed in full screen and the remaining players are hidden.

7. The method according to any one of claims 1 to 4, characterized in that: The step of responding to the first frame selection instruction and determining, from a plurality of three-dimensional labels of the digital twin world at the first perspective, three-dimensional labels falling within a frame selection area of ​​the first frame selection instruction to obtain a first label set includes: Projecting each of the plurality of three-dimensional tags to a screen coordinate system to obtain a plurality of projection images; constructing an outer bounding box for each of the plurality of projection images to obtain a plurality of outer bounding boxes; In response to the first frame selection instruction, according to each outer bounding box of the plurality of outer bounding boxes and the frame selection area of ​​the first frame selection instruction, a three-dimensional label falling within the frame selection area of ​​the first frame selection instruction is determined to obtain a first label set.

8. The method according to claim 7, characterized in that The step of responding to the first frame selection instruction and determining, according to each outer bounding box of the plurality of outer bounding boxes and the frame selection area of ​​the first frame selection instruction, a three-dimensional label falling within the frame selection area of ​​the first frame selection instruction to obtain a first label set includes: In response to the first frame selection instruction, determining a plurality of separation axes according to the frame selection area of ​​the first frame selection instruction; For each outer bounding box of the plurality of outer bounding boxes, determining projections of the outer bounding box and the frame selection area of ​​the first frame selection instruction on each separation axis; When there is at least one intersecting separation axis among the multiple separation axes, it is determined that the label corresponding to the outer bounding box falls within the frame selection area of ​​the first frame selection instruction, and the projections of the outer bounding box and the frame selection area of ​​the first frame selection instruction on the intersecting separation axis overlap.

9. A display device for monitoring video, characterized in that: include: A display module, used to display a digital twin world from a first perspective on a user interface, wherein the digital twin world is a virtual representation of a real physical world, and the digital twin world has a plurality of three-dimensional labels, wherein the three-dimensional labels represent cameras in the physical world; a processing module, configured to, in response to a first frame selection instruction, determine, from a plurality of three-dimensional labels of the digital twin world at the first perspective, three-dimensional labels that fall within a frame selection area of ​​the first frame selection instruction to obtain a first label set; A pulling module, used to pull the video stream of the camera corresponding to each three-dimensional tag in the first tag set to obtain multiple video streams; The playing module is used to play multiple video streams on the user interface to display the multiple video streams.

10. An electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images