Image monitoring system, method and computer readable medium using digital twin model

Abstract

The invention discloses an image monitoring system, method and computer readable medium using digital avatar model. A data linking module creates a digital avatar model based on camera information of a camera, a simulation rendering module calculates a project polygon range of the camera based on attributes of the digital avatar model, and then calculates UV mapping coordinates used for rendering in a digital avatar platform based on the project polygon range, an image width and an image height of the camera, so as to project an image stream of the camera onto a 3D model of the digital avatar platform based on the UV mapping coordinates for overlay rendering to simulate overlay effect of a real image of the camera onto the digital avatar platform. When a spatial positioning module receives an alarm point of a concern event, it can notify the digital avatar platform of an alarm, and also activate surrounding effective cameras according to the alarm point of the concern event to obtain relevant surveillance information.

Description

Technical Field

[0001] This invention relates to an image surveillance technology, and more particularly to an image surveillance system, method, and computer-readable medium using a digital clone model. Prior Technology

[0002] Due to the significant increase in the number of cameras in various fields and the progress in artificial intelligence (AI) image recognition, there are numerous related systems and sites. However, apart from AI technology, most inspection platforms remain at the level of text and two-dimensional descriptions.

[0003] Existing technologies propose a full-time and space-time stereoscopic visualization method, but this method is very time-consuming and labor-intensive.

[0004] Furthermore, the aforementioned existing technologies cannot establish a digital clone model based on the camera information, nor can they allow the digital clone platform to simulate the camera's viewing range, nor can they project the camera's image stream onto the digital clone platform. Moreover, they cannot quickly locate and automatically activate nearby effective cameras based on the alarm points of events of interest to the user, thus making it difficult to monitor events of interest to the user in real time.

[0005] Therefore, how to provide an innovative video surveillance technology to solve any of the above problems, or to provide related systems and methods, has become a major research topic for those skilled in the art. Summary of the Invention

[0006] The image monitoring system using a digital avatar model as described in this invention includes: a data connection module, which establishes a digital avatar model of the camera based on camera information; an analog rendering module and a digital avatar platform. The analog rendering module is communicatively connected to both the data connection module and the digital avatar platform. The analog rendering module calculates a digital array of the camera's illumination polygon range based on the attributes of the camera's digital avatar model. Then, the analog rendering module calculates the UV mapping coordinates used for rendering in the digital avatar platform based on the digital array of the camera's illumination polygon range, the camera's image width, and the camera's image height. The simulation rendering module uses UV mapping coordinates to create image textures from the camera's image stream and projects them onto the 3D model of the digital avatar platform for overlay rendering, simulating the overlay effect of the camera's real image on the digital avatar platform; and a spatial positioning module, which is connected to the digital avatar platform, notifies the digital avatar platform of an alarm when it receives a push notification of an alarm point related to an event of interest to the user. The spatial positioning module then activates nearby cameras based on the alarm point of the event of interest to obtain monitoring information of the event of interest to the user.

[0007] The image monitoring method using a digital clone model described in this invention includes: a data connection module establishing a digital clone model of the camera based on camera information; an analog rendering module calculating a digital array of the camera's illumination polygon range based on the attributes of the camera's digital clone model; the analog rendering module then calculating UV mapping coordinates used for rendering in a digital clone platform based on the digital array of the camera's illumination polygon range, the camera's image width, and the camera's image height; the analog rendering module then using the UV mapping coordinates to create image textures from the camera's image stream and projecting them onto the three-dimensional model of the digital clone platform for overlay rendering, simulating the overlay effect of the camera's real image being presented on the digital clone platform; and when a spatial positioning module receives a push message about an alarm point related to an event of interest to the user, the spatial positioning module notifies the digital clone platform of an alarm, allowing the spatial positioning module to activate nearby cameras based on the alarm point of the event of interest to obtain monitoring information about the event of interest to the user.

[0008] The computer-readable medium of the present invention is applied in a computing device or computer, stores instructions, and is executed by a processor and memory, so as to perform the above-mentioned image monitoring method using a digital clone model when the computer-readable medium is executed.

[0009] Therefore, the present invention provides an innovative video surveillance system, method, and computer-readable medium using a digital clone model. The system automatically or effectively creates a digital clone model of the camera based on the camera information from the camera via a data link module. This allows the digital clone platform to not only simulate the camera's viewing range but also to project the camera's video stream onto the digital clone platform. This facilitates, for example, efficient comparison of the camera's installation effect or data accuracy, and also identifies blind spots on the user's end.

[0010] Furthermore, the digital clone platform of this invention can simulate the position and projection range of a camera, and can also quickly locate and automatically activate surrounding cameras based on the alarm points of events of interest to the user (such as alarm events / emergency events).

[0011] In addition, when an alarm for a user-related event is triggered, the spatial positioning module can automatically locate the surrounding valid cameras from a large amount of managed camera information based on the location and range of the event. It can also automatically determine the validity of the cameras and activate and inspect the valid cameras, eliminating the need for manual judgment on the user's end, thus facilitating real-time monitoring of events of interest to the user.

[0012] To make the above-described features and advantages of the present invention more apparent and understandable, specific embodiments are described below in conjunction with the accompanying drawings. Additional features and advantages of the invention will be set forth in part in the following description, and such features and advantages will be partly apparent from the description or may be learned by practice of the invention. It should be understood that both the foregoing general description and the following detailed description are illustrative and explanatory, and are not intended to limit the scope of the invention. Simple Explanation of the Diagram

[0013] Figure 1 is a schematic diagram of the architecture of the image monitoring system using a digital clone model according to the present invention.

[0014] Figure 2 is a flowchart illustrating the operation of the data linking module, the simulation rendering module, and the spatial positioning module in the image monitoring system and method using a digital clone model as described in this invention.

[0015] Figures 3A and 3B are front and side views, respectively, of the projection effect of the analog rendering module projecting the camera's image stream onto the digital avatar platform in the image monitoring system and method using the digital avatar model described in this invention.

[0016] Figure 4 is a flowchart illustrating the method for quickly searching a list of low / medium / high-efficiency cameras using a digital clone model in the image surveillance system and method using a digital clone model according to the present invention. Implementation

[0017] The following describes the implementation of the present invention through specific embodiments. Those skilled in the art can understand other advantages and effects of the present invention from the content disclosed in this specification, and can therefore implement or use it through other different specific equivalent embodiments.

[0018] Figure 1 is a schematic diagram of the architecture of the image monitoring system 1 using a digital clone model according to the present invention.

[0019] As shown in Figure 1, the video surveillance system 1 using digital clone models may include a data link module 10 that communicates with each other, at least one (such as multiple or a large number) camera 20, a digital clone platform 30, an analog rendering module 40, a spatial positioning module 50, and a spatial index database 60, etc., and the data link module 10 can create digital clone models 11 of the camera 20, etc.

[0020] In one embodiment, the data link module 10, the digital clone platform 30, the simulation rendering module 40, the spatial positioning module 50, and the spatial index database 60 can all be communicatively connected to the camera 20, etc. The data link module 10 can further be communicatively connected to the digital clone platform 30, the simulation rendering module 40, the spatial index database 60, etc. The digital clone platform 30 can further be communicatively connected to the simulation rendering module 40, the spatial positioning module 50, etc. The simulation rendering module 40 and the spatial positioning module 50 can further be communicatively connected to the spatial index database 60, etc.

[0021] In one embodiment, the data connection module 10 can be a data connection component (chip / circuit), data connection software (program), etc.; the digital avatar model 11 can be a virtual avatar model, etc.; and the camera 20 can be a network camera, monitor, video recorder, or other physical camera. The digital avatar platform 30 can be a virtual avatar platform, digital avatar system, etc., such as Bentley Systems CesiumJS, NVIDIA Omniverse, etc. The simulation rendering module 40 can be a simulation rendering component (chip / circuit), simulation rendering software (program), etc.; the spatial positioning module 50 can be a spatial positioning component (chip / circuit), spatial positioning software (program), etc.; and the spatial index database 60 can be a spatial index data storage device, spatial index data hard disk, spatial index data server, etc. The user terminal 70 can be the user, monitor, manager, event observer and / or the electronic device used (such as smartphone / tablet / laptop / personal computer, etc.), the event of concern 71 can be various alarm events or emergency events such as fire, flood, wind disaster, earthquake, landslide, traffic accident, gas explosion, air crash, infectious disease, etc., and the alarm point 72 can be alarm coordinates or alarm location, etc.

[0022] In this invention, "at least one" means one or more (e.g., one, two, or three or more), "plural" or "multiple" means two or more (e.g., two, three, four, or ten or more), and "communication connection" means mutual communication or connection through various forms such as data, signal, electrical, wired (e.g., wired network), or wireless (e.g., wireless network). However, this invention is not limited to those mentioned in the various embodiments.

[0023] First, the data link module 10 can first establish the digital clone model 11 based on the camera information 21 (such as camera attributes) of the camera 20 or the attributes of the digital clone model 11, and the camera information 21 (such as camera attributes) or the attributes of the digital clone model 11 can be shown in the following list 1.

[0024] For example, camera information 21 (such as camera attributes) may include various information or attributes of camera 20, such as attitude, coordinates, viewing azimuth, viewing distance, etc.

[0025] Table 1: Camera information of camera 20 or attributes of digital clone model 11.

[0026] [1] Camera 20 number: Data link module 10 can name and store a unique number Cam_ID (such as an identification code) for each camera 20 to identify different cameras 20.

[0027] [2] Name of camera 20: The data link module 10 can set a text-type name Cam_Name for each camera 20 to describe the function of the camera 20 for reference by the user terminal 70.

[0028] [3] Camera 10 connection (e.g., streaming connection): The data connection module 10 can set a text-type connection Cam_Url (e.g., streaming connection) for each camera 20 to obtain the image streaming 24 of the camera 20 through the connection Cam_Url of the camera 10.

[0029] [4] Longitude coordinates of camera 20: The data link module 10 can set a digital longitude coordinate Cam_PosX for each camera 20 to calculate the irradiated polygon range Cam_ProjArea of ​​camera 20, and also to calculate the effectiveness of related cameras 20 when the alarm of the user terminal 70's attention event 71 is triggered.

[0030] [5] Latitude coordinates of camera 20: Data link module 10 can set digital latitude coordinates Cam_PosY for each camera 20 to calculate the irradiated polygon range Cam_ProjArea of ​​camera 20, and also to calculate the effectiveness of related cameras 20 when the alarm of the user terminal 70's attention event 71 is triggered.

[0031] [6] Camera 20 height: The data link module 10 can set a digital height Cam_Altitude for each camera 20 to calculate the illumination polygon range Cam_ProjArea of ​​the camera 20, and also to calculate the effectiveness of the relevant camera 20 when the alarm of the user terminal 70's attention event 71 is triggered.

[0032] [7] Azimuth angle of camera 20: The data connection module 10 can set a digital azimuth angle Cam_Heading for each camera 20 to calculate the illumination polygon range Cam_ProjArea of ​​camera 20, and also to calculate the effectiveness of related cameras 20 when the alarm of the user terminal 70's attention event 71 is triggered.

[0033] [8] Roll angle of camera 20: The data link module 10 can set a digital roll angle Cam_Roll for each camera 20 to calculate the illumination polygon range Cam_ProjArea of ​​camera 20, and also to calculate the effectiveness of related cameras 20 when the alarm of the user terminal 70's attention event 71 is triggered.

[0034] [9] Pitch angle of camera 20: The data link module 10 can set a digital pitch angle Cam_Pitch for each camera 20 to calculate the illumination polygon range Cam_ProjArea of ​​camera 20, and also to calculate the effectiveness of related cameras 20 when the alarm of the user terminal 70's attention event 71 is triggered.

[0035]

[10] Illumination polygon range of camera 20: The data link module 10 can set a digital array type illumination polygon range Cam_ProjArea for each camera 20 as a cache, so as to store the calculated illumination polygon range Cam_ProjArea of ​​camera 20 through the cache (such as cache memory), which helps to reduce the number of times the illumination polygon range Cam_ProjArea of ​​camera 20 is calculated in real time, and can also calculate the UV mapping coordinates 41 corresponding to the camera image 22 of camera 20.

[0036]

[11] Image width of camera 20: Data link module 10 can set a digital image width Cam_ImgWidth for each camera 20 to calculate the UV mapping coordinates 41 corresponding to the camera image 22 of camera 20.

[0037]

[12] Image height of camera 20: Data link module 10 can set a digital image height Cam_ImgHeight for each camera 20 to calculate the UV mapping coordinates 41 corresponding to the camera image 22 of camera 20.

[0038]

[13] The line-of-sight distance of the camera 20: The data link module 10 can set a digital line-of-sight distance Cam_Distance for each camera 20 to calculate the effectiveness of the relevant camera 20 when the alarm of the user terminal 70’s attention event 71 is triggered.

[0039] In one embodiment, UV mapping coordinates 41 can be the process of projecting (projecting) a two-dimensional image onto a three-dimensional surface for texture mapping during the creation of a three-dimensional model 32, and U and V can represent coordinates or coordinate axes. Furthermore, the event of interest 71 can be various alarm events or emergency events such as fire, flood, windstorm (e.g., typhoon), earthquake, landslide, traffic accident (e.g., car crash), gas explosion, air crash, infectious disease (e.g., infectious pneumonia).

[0040] Figure 2 is a flowchart illustrating the operation of the data connection module 10, the simulation rendering module 40, and the spatial positioning module 50 in the image monitoring system 1 and method using the digital avatar model shown in Figure 1 of the present invention. Figures 3A and 3B are front and side views, respectively, of the projection effect of the simulation rendering module 40 projecting (projecting) the image stream 24 of the camera 20 onto the digital avatar platform 30 in the image monitoring system 1 and method using the digital avatar model shown in Figure 1 of the present invention.

[0041] The image surveillance system 1 and method using a digital clone model described in this invention may include:

[0042] [1] The data link module 10 creates a digital clone model 11 (such as a virtual clone model) of the camera 20 based on the camera information 21 (such as camera attributes or attribute data) of the camera 20. The data link module 10 can also update the camera information 21 (such as camera attributes) of the camera 20 in real time, such as the attitude, coordinates, viewing azimuth angle, viewing distance, etc.

[0043] [2] The simulation rendering module 40 simulates the illumination polygon range Cam_ProjArea of ​​the camera 20 based on the camera information 21 of the camera 20 (such as camera attributes) or the attributes of the digital clone model 11, and calculates the UV mapping coordinates 41 corresponding to the camera image 22 of the camera 20 or used for rendering in the digital clone platform 30. Then, the simulation rendering module 40 renders and projects the geospatial data 31 (such as the three-dimensional surface of the terrain) of the digital clone platform 30 and overlays it to present a more comprehensive understanding of the monitoring area or the area of ​​interest of the user terminal 70 (such as the monitor).

[0044] [3] The spatial positioning module 50 establishes a data structure by using a spatial indexing method (such as a spatial indexing algorithm) to establish the camera coordinates 23 of the added camera 20, and provides quick positioning and automatic activation of the surrounding effective cameras 20 based on the alarm point 72 (such as alarm coordinates / alarm location) of the event 71 of interest to the user terminal 70, so as to quickly obtain the relevant monitoring information of the event 71 of interest to the user terminal 70.

[0045] For example, geospatial data 31 may include data (information) of various three-dimensional surfaces of landforms such as terrain models, building models, and oblique photogrammetry models. "Rendering" may refer to the process of generating images from models using software in computer graphics.

[0046] 1. Data Link Module 10: As shown in step S11 of Figure 2, a digital clone model 11 of camera 20 is created. That is, the data link module 10 can use the camera information 21 of camera 20 (such as camera attributes) or the attributes of digital clone model 11 to create a digital clone model 11 of camera 20.

[0047] In one embodiment, the user terminal 70 can use the user interface of the image monitoring system 1 to display the following information about the camera 20: [1] camera description (such as the camera number Cam_Id and name Cam_Name), [2] camera position (such as the longitude coordinates Cam_PosX, latitude coordinates Cam_PosY and altitude Cam_Altitude), [3] camera attitude (such as the azimuth angle Cam_Heading, roll angle Cam_Roll and pitch angle Cam_Pitch), [4] camera parameters (such as the connection Cam_Url and image width Cam_I of the camera 10). The camera information 21, such as mgWidth, image height Cam_ImgHeight, and viewing distance Cam_Distance, is used to build a digital clone model 11. The data link module 10 uses multiple camera information 21 (such as camera position, camera posture, and camera parameters such as Cam_Url of camera 20) to automatically calculate [5] the illumination polygon range Cam_ProjArea of ​​camera 20 (such as the projection range 34 of camera 20). Then, the data link module 10 calculates the spatial index structure (such as R-tree) based on the minimum bounding rectangle of the illumination polygon range Cam_ProjArea of ​​camera 20. The data link module 10 then uses the spatial index structure to insert the above [1] to [5] information into the spatial index database 60 for storage (saving), so as to complete the addition of the digital clone model 11 of camera 20.

[0048] II. Simulation Rendering Module 40: This module can use the data linking module 10 to store (save) camera information 21 (such as camera parameters) 21 (e.g., the illumination polygon range Cam_ProjArea and connection Cam_Url, such as streaming links) of the camera 20 in the spatial index database 60. It then uses homography to calculate the UV mapping coordinates 41 used for rendering in the digital avatar platform 30. For example, "homography" can be a reversible transformation from a real projective plane to a projective plane in geometry, or the transformation between any two sets of coordinates calculated using a two-dimensional matrix.

[0049] Furthermore, as shown in step S12 of Figure 2 and Figures 3A to 3B, the image of the camera 20 is projected onto the 3D model 32 (such as a 3D scene). That is, the simulation rendering module 40 can build image textures from the image stream 24 (or streaming image) of the camera 20 and then project it onto the digital clone platform 30 with the 3D model 32 (such as a 3D scene) to simulate the overlay effect of the real image 33 of the camera 20 onto the digital clone platform 30.

[0050] In one embodiment, the projection effect of the simulation rendering module 40 can be seen in the front view of Figure 3A, where the real camera image 33 of the camera 20 (such as the real camera image 22) is projected onto the 3D model 32 (such as a 3D scene) of a simple monochrome object (such as a building), to achieve the overlay effect of the real camera image 33 simulating the digital avatar platform 30. The side view of Figure 3B is a perspective from another angle, which more clearly shows that the real camera image 33 of the camera 20 is overlaid onto the 3D model 32 (such as a 3D scene) of the simple monochrome object (such as a building). This simulation rendering module 40 can check the update of the image stream 24 of the camera 20 that is turned on for each rendering. If the image stream 24 of the camera 20 has been updated, the simulation rendering module 40 will update the image texture of the established camera 20, but will not recalculate the UV mapping coordinates 41.

[0051] During rendering on the digital avatar platform 30, the simulation rendering module 40 pre-calculates the digital array of the illumination polygon range Cam_ProjArea of ​​the camera 20 based on the attributes of the digital avatar model 11 of the camera 20 each time the digital avatar platform 30 renders a scene (for example, 30 to 60 times per second under smooth conditions). Then, the simulation rendering module 40 uses the digital array of the illumination polygon range Cam_ProjArea of ​​the camera 20, the image width Cam_ImgWidth of the camera 20, and the image height Cam_ImgHeight of the camera 20 as a reference. The UV mapping coordinates 41 used for rendering in the digital clone platform 30 are calculated using homography. The simulation rendering module 40 then uses the UV mapping coordinates 41 to create image textures from the image stream 24 (or streaming image) of the camera 20 and projects them onto the scene terrain of the 3D model 32 of the digital clone platform 30 for overlay rendering. As shown in the front view of Figure 3A and the side view of Figure 3B, the simulation rendering module 40 projects the image of the camera 20 onto the 3D model 32 of the virtual object (such as a virtual building) of the digital clone platform 30, so that the real image 33 of the camera 20 is presented to the digital clone platform 30 for overlay effect.

[0052] 3. Spatial Positioning Module 50: Provides spatial calculation and positioning range calculation functions for use by the digital clone platform 30. That is, as shown in step S13 of Figure 2, the spatial positioning module 50 can provide or calculate three levels of effective camera suggestion lists (such as the first level of low-effectiveness camera suggestion list 51, the second level of medium-effectiveness camera suggestion list 52, and the third level of high-effectiveness camera suggestion list 53) based on the effectiveness of the camera 20, and can also calculate the positioning range 54.

[0053] [1] The spatial positioning module 50 can calculate a first-level list of low-efficiency cameras 51. That is, the spatial positioning module 50 can perform a spatial distance search on the spatial index database 60 based on the location information of the alarm point 72 of the event of interest 71 of the user terminal 70 (such as the latitude and longitude coordinates of the alarm point 72) to obtain a plurality of cameras 20 with Cam_ID within the maximum distance (unit: meters). The spatial positioning module 50 then uses the list of cameras 20 with Cam_ID as the first-level list of low-efficiency cameras 51 and sends it back to the digital clone platform 30. For example, the maximum distance can be preset to 100 meters, or it can be adjusted according to the actual needs of the user terminal 70.

[0054] [2] The spatial positioning module 50 can calculate the effective camera suggestion list 52 in the second layer. That is, the spatial positioning module 50 can verify the viewing distance Cam_Distance and azimuth angle Cam_Heading of the multiple cameras 20 listed in the low-effective camera suggestion list 51 of the first layer, so that the spatial positioning module 50 can quickly calculate the multiple numbered Cam_ID of the multiple cameras 20 within the viewing distance Cam_Distance and the azimuth angle Cam_Heading within a predetermined positive and negative degree (such as positive and negative 45 degrees), so that the spatial positioning module 50 can use the number list of multiple numbered Cam_ID cameras 20 as the effective camera suggestion list 52 in the second layer to send back to the digital clone platform 30.

[0055] [3] The spatial positioning module 50 can calculate the high-efficiency camera recommendation list 53 of the third level. That is, the spatial positioning module 50 can perform spatial geometric calculation (geometric inclusion operation) on the location information of the illumination polygon area Cam_ProjArea of ​​the multiple numbered Cam_ID of the camera 20 listed in the effective camera recommendation list 52 of the second level and the alarm point 72 of the user terminal 70. Then, the spatial positioning module 50 uses the numbered Cam_ID of the camera 20 included in the result of the spatial geometric calculation (geometric inclusion operation) as the high-efficiency camera recommendation list 53 of the third level and sends it back to the digital clone platform 30.

[0056] [4] The spatial positioning module 50 can calculate the positioning range 54. That is, the spatial positioning module 50 can calculate the minimum bounding rectangle as the positioning range 54 according to the camera coordinates 23 (such as the longitude coordinates Cam_PosX and latitude coordinates Cam_PosY of camera 20) of the effective camera suggestion list of the specified level (such as the low effective camera suggestion list 51 of the first level, the medium effective camera suggestion list 52 of the second level, or the high effective camera suggestion list 53 of the third level), and then the spatial positioning module 50 transmits the positioning range 54 back to the digital clone platform 30.

[0057] In other words, the image monitoring system 1 using the digital clone model described in this invention can be applied to rapid location and presentation when an alarm occurs. For example, in the event of a disaster alarm, after the spatial positioning module 50 obtains the latitude and longitude (longitude / latitude) and altitude values ​​of the event 71 of interest to the user terminal 70, it can search for three levels of effective camera suggestion lists (camera suggestion lists) such as the low-effectiveness camera suggestion list 51, the medium-effectiveness camera suggestion list 52, and the high-effectiveness camera suggestion list 53 according to the pre-established spatial index database 60.

[0058] [1] First level: The spatial positioning module 50 can quickly search for the specified alarm and the first level of low-efficiency camera suggestion list 51 within the specified range through spatial indexing methods (such as spatial indexing algorithms). The technical feature of the first level of low-efficiency camera suggestion list 51 is that it can quickly display a large amount of relevant camera information 21, but the efficiency is low, and cameras 20 outside the visible distance and at different azimuth angles will also be included.

[0059] [2] Second level: The spatial positioning module 50 can further calculate based on the visible distance Cam_Distance (unit: meters) and azimuth angle Cam_Heading of the camera 20 according to the low-efficiency camera suggestion list 51 of the first level to filter out the effective camera suggestion list 52 in the second level. The technical feature of the effective camera suggestion list 52 in the second level is the balance between calculation speed and effectiveness. First, the number of cameras 20 that need to be calculated is greatly reduced through the low-efficiency camera suggestion list 51 of the first level, and then the calculation of the visible distance and azimuth angle with low computing power requirements is performed. For example, compared with the low-efficiency camera suggestion list 51 of the first level, the effective camera suggestion list 52 in the second level can eliminate up to 75% of the ineffective cameras 20, and the effectiveness averages as high as 90%.

[0060] [3] Third level: The spatial positioning module 50 can further perform spatial geometric calculations based on the illumination polygon range Cam_ProjArea of ​​the camera 20 according to the effective camera suggestion list 52 in the second level, so as to retain the high-efficiency camera suggestion list 53 of the third level that meets the range conditions. The effectiveness of this stage (the high-efficiency camera suggestion list 53 of the third level) is 100% when the camera information 21 of the camera 20 is correct, but the calculation is more complicated and the calculation speed will be affected by the shape complexity of the illumination polygon range Cam_ProjArea of ​​the camera 20.

[0061] For example, the present invention can be preset to a low-efficiency camera suggestion list 51 at the first level, and allow the user terminal 70 to manually switch between the low-efficiency camera suggestion list 51, the medium-efficiency camera suggestion list 52, or the high-efficiency camera suggestion list 53 according to different situations to obtain more camera information 21. If the values ​​of the camera information 21 are not accurate enough, resulting in the medium-efficiency camera suggestion list 52 being more accurate, the user terminal 70 can manually switch back to the medium-efficiency camera suggestion list 52.

[0062] Figure 4 is a flowchart illustrating the method for quickly searching a list of low / medium / high-efficiency cameras using the digital clone model 11 in the image surveillance system 1 and method using the digital clone model 11 described in this invention, including the following steps S21 to S26, and is explained together with reference to Figure 1.

[0063] As shown in steps S21 and S22, alarm coordinates and notifications of an alarm are received. That is, when the spatial positioning module 50 receives a push message about the alarm point 72 (such as alarm coordinates / alarm location) of the user terminal 70's attention event 71, the spatial positioning module 50 will notify the digital clone platform 30 that there is an alarm. After receiving the notification from the spatial positioning module 50, the digital clone platform 30 can call the spatial positioning module 50 to request the calculation of the first-level low-efficiency camera suggestion list 51.

[0064] As shown in steps S23, S24, and S25, the system requests information on the camera group at a specified level, calculates the camera group at the specified level, and calculates the positioning range 54 of the camera group at the specified level. That is, after receiving the message from the digital clone platform 30 requesting the calculation of the first-level low-efficiency camera suggestion list 51, the spatial positioning module 50 will perform a spatial distance search on the spatial index database 60 based on the location information of the alarm point 72 of the user terminal 70's attention event 71 (such as the latitude and longitude coordinates of the alarm point 72) to obtain the multiple cameras 20 with the number Cam_Id within the maximum distance (unit: meters), so that the list of multiple cameras 20 with the number Cam_ID is used as the first-level low-efficiency camera suggestion list 51. For example, the maximum distance can be preset to 100 meters, or it can be adjusted according to the actual needs of the user terminal 70. Next, the spatial positioning module 50 can calculate the minimum bounding rectangle as the positioning range 54 according to the camera coordinates 23 (such as the longitude coordinates Cam_PosX and latitude coordinates Cam_PosY of camera 20) listed in the low-efficiency camera suggestion list 51 of the first level, so that the spatial positioning module 50 can transmit the low-efficiency camera suggestion list 51 and the positioning range 54 of the first level back to the digital clone platform 30.

[0065] As shown in step S26, information about the camera group at the specified level is notified. That is, after the digital clone platform 30 receives the list of low-efficiency cameras 51 and the positioning range 54 of the first level returned by the spatial positioning module 50, it can zoom to the positioning range 54 for inspection to simulate the inspection range of the camera 20. It can also call the simulation rendering module 40 to project the real camera images 33 of all cameras 20 listed in the list of low-efficiency cameras 51 of the first level onto the scene terrain of the 3D model 32 for overlay rendering to simulate the effect of the real scene. If the effectiveness of the camera effect is insufficient, the digital clone platform 30 can call the spatial positioning module 50 again to request the calculation of the list of effective cameras 52 in the second level.

[0066] As shown in steps S23, S24, and S25, after receiving the message from the digital clone platform 30 requesting the calculation of the effective camera suggestion list 52 in the second layer, the spatial positioning module 50 quickly calculates the effective camera suggestion list 52 in the second layer from the low-effective camera suggestion list 51 in the first layer according to the visible distance Cam_Distance and azimuth angle Cam_heading of the camera 20. Then, the spatial positioning module 50 calculates the minimum bounding rectangle as the positioning range 54 according to the camera coordinates 23 (such as the longitude coordinates Cam_PosX and latitude coordinates Cam_PosY of the camera 20) listed in the effective camera suggestion list 52 in the second layer, so that the spatial positioning module 50 can send the effective camera suggestion list 52 in the second layer and the positioning range 54 back to the digital clone platform 30.

[0067] As shown in step S26, after receiving the second-level effective camera suggestion list 52 and positioning range 54 returned by the spatial positioning module 50, the digital clone platform 30 can zoom to the positioning range 54 for inspection to simulate the inspection range of the camera 20. It can also call the simulation rendering module 40 to project the real camera images 33 of all cameras 20 listed in the second-level effective camera suggestion list 52 onto the scene and terrain of the 3D model 32 for overlay rendering to simulate the effect of the real scene. If the effectiveness of the camera effect is insufficient, the digital clone platform 30 can call the spatial positioning module 50 again to request the calculation of the third-level highly effective camera suggestion list 53.

[0068] As shown in steps S23, S24, and S25, after receiving the message from the digital clone platform 30 requesting the calculation of the high-efficiency camera suggestion list 53 for the third layer, the spatial positioning module 50 quickly calculates the high-efficiency camera suggestion list 53 for the third layer based on the visible distance Cam_Distance of the camera 20 from the high-efficiency camera suggestion list 52 in the second layer. Then, the spatial positioning module 50 calculates the minimum bounding rectangle as the positioning range 54 based on the camera coordinates 23 (such as the longitude coordinates Cam_PosX and latitude coordinates Cam_PosY of the camera 20) of the high-efficiency camera suggestion list 53 for the third layer, so that the spatial positioning module 50 can send the high-efficiency camera suggestion list 53 for the third layer and the positioning range 54 back to the digital clone platform 30.

[0069] As shown in step S26, after the digital clone platform 30 receives the third-level high-efficiency camera suggestion list 53 and positioning range 54 returned by the spatial positioning module 50, it can zoom to the positioning range 54 for inspection to simulate the inspection range of the camera 20. It can also call the simulation rendering module 40 to project the real camera images 33 of all cameras 20 listed in the third-level high-efficiency camera suggestion list 53 onto the scene and terrain of the 3D model 32 for overlay rendering to simulate the effect of the real scene. If the camera effect is reduced in effectiveness due to the camera information 21 not being completely normal, the spatial positioning module 50 can be requested again for inspection of the second-level effective camera suggestion list 52.

[0070] Furthermore, the present invention also provides a computer-readable medium for an image surveillance method using a digital avatar model, which is applied in a computing device or computer having a processor and memory. The computer-readable medium stores instructions and can be executed by the computing device or computer through the processor and memory to perform the aforementioned image surveillance method using a digital avatar model. In another embodiment, this computer-readable medium is a non-transitory computer-readable storage medium.

[0071] In one embodiment, the processor may be a central processing unit (CPU), graphics processing unit (GPU), microprocessor (MPU), microcontroller (MCU), artificial intelligence (AI) processor, etc., and the memory may be random access memory (RAM), read-only memory (ROM), flash memory, memory card, hard disk (such as cloud / network / external hard disk), optical disk, USB flash drive, database, etc., and the computing device or computer may be a computer, smartphone, tablet computer, personal computer, notebook computer, desktop computer, server (such as cloud / remote / network / AI server), etc.

[0072] In one embodiment, the modules can be software, hardware, or firmware; if they are hardware, they can be processing units, processors, computers, or servers with data processing and computing capabilities; if they are software or firmware, they can include instructions executable by processing units, processors, computers, or servers, and can be installed on the same hardware device or distributed across different multiple hardware devices.

[0073] In summary, the image monitoring system, method, and computer-readable medium using digital clone models described in this invention have at least the following features, advantages, or technical effects.

[0074] 1. The data linking module 10 of the present invention can automatically or effectively create a digital clone model 11 of the camera 20 based on the camera information 21 of the camera 20.

[0075] Second, this invention enables the digital clone platform 30 to not only simulate the viewing range of the camera 20, but also to project the image stream 24 of the camera 20 onto the digital clone platform 30. This allows for efficient comparison of the three-dimensional surface of the geospatial data 31 (such as terrain model, building model, oblique photogrammetry model, etc.) projected from the camera 20 onto the digital clone platform 30 with the camera 20 to determine the installation effect or data accuracy of the camera 20. It also helps to identify the visual blind spots of the user terminal 70.

[0076] Third, the digital clone platform 30 of the present invention can simulate the position and projection range 34 of the camera 20, and can also quickly locate and automatically turn on the camera 20 within the monitoring range based on the alarm point 72 (such as alarm coordinates / alarm location) of the event 71 of the user terminal 70. With the assistance of geospatial data 31 (such as three-dimensional surface of the landform) and image overlay in the digital clone platform 30, the image monitoring area of ​​the camera 20 can be understood more comprehensively and specifically. It can also effectively examine the cause of the problem and eliminate blind spots due to insufficient planar image information.

[0077] Fourth, when an alarm for a user-end 70’s attention event 71 (such as an alarm event / emergency event) is triggered, the spatial positioning module 50 can automatically locate the surrounding valid cameras 20 from the camera information 21 of a large number of managed cameras based on the location and range of the attention event 71. It can also automatically determine the validity of the cameras 20 to activate and inspect the valid cameras 20, without requiring manual judgment from the user-end 70, thus facilitating real-time monitoring of the user-end 70’s attention event 71.

[0078] Fifth, the present invention can provide three levels of camera suggestion list 51 (low effectiveness), camera suggestion list 52 (medium effectiveness), and camera suggestion list 53 (high effectiveness) according to the effectiveness of the camera 20, so that the user terminal 70 can quickly switch and learn more (additional) information about the camera 20 according to different situations.

[0079] VI. The spatial positioning module 50 of the present invention can quickly search for a list 51 of low-efficiency cameras of the first level within a specified alarm and a specified range through a spatial indexing method (such as a spatial indexing algorithm), so as to calculate a large amount of relevant camera information 21 that can be displayed in real time.

[0080] 7. The spatial positioning module 50 of the present invention can calculate again based on the visible distance Cam_Distance (unit: meters) and azimuth angle Cam_Heading of the camera 20 according to the low-efficiency camera suggestion list 51 of the first level to filter out the effective camera suggestion list 52 of the second level. This is beneficial to achieving a balance between calculation speed and effectiveness, and can also effectively eliminate some of the ineffective cameras 20 and improve the average effectiveness.

[0081] 8. The spatial positioning module 50 of the present invention can perform spatial geometric calculations again based on the illumination polygon range Cam_ProjArea of ​​the camera 20 according to the effective camera suggestion list 52 in the second level, so as to efficiently retain the high-effectiveness camera suggestion list 53 of the third level that meets the range conditions, and can also make the effectiveness reach 100% when the camera information 21 of the camera 20 is correct.

[0082] The above embodiments are merely illustrative of the principles, features, and effects of the present invention and are not intended to limit the scope of implementation of the present invention. Anyone skilled in the art can modify and alter the above embodiments without departing from the spirit and scope of the present invention. Any equivalent changes and modifications made using the content disclosed in this invention should still be covered by the claims. Therefore, the scope of protection of this invention should be as listed in the claims.

[0083]

[0084] 1: Video surveillance system

[0085] 10: Data Link Module

[0086] 11: Digital Clone Model

[0087] 20: Camera

[0088] 21: Camera Information

[0089] 22: Camera footage

[0090] 23: Camera Coordinates

[0091] 24: Video Streaming

[0092] 30: Digital Avatar Platform

[0093] 31: Geospatial Data

[0094] 32: Three-dimensional model

[0095] 33: Actual Camera Footage

[0096] 34: Projection range

[0097] 40: Simulation Rendering Module

[0098] 41: UV mapping coordinates

[0099] 50: Spatial Positioning Module

[0100] 51: Recommended list of low-efficiency cameras

[0101] 52: Recommended list of medium-efficiency cameras

[0102] 53: Recommended list of high-efficiency cameras

[0103] 54: Positioning Range

[0104] 60: Spatial Index Database

[0105] 70: User End

[0106] 71: Focus on the event

[0107] 72: Alarm Point

[0108] S11 to S13: Steps

[0109] S21 to S26: Steps

Claims

1. A video surveillance system using a digital clone model, comprising: A data link module creates a digital clone model of the camera based on camera information; a simulation rendering module and a digital clone platform are connected to the data link module and the digital clone platform respectively. The simulation rendering module calculates the digital array of the camera's illumination polygon range based on the attributes of the digital clone model. Then, based on the digital array of the camera's illumination polygon range, the camera's image width, and the camera's image height, the simulation rendering module calculates the UV mapping coordinates used for rendering in the digital clone platform. The simulation rendering module then uses these UV mapping coordinates to create image textures from the camera's image stream and projects them onto the 3D model of the digital clone platform for overlay rendering, thus simulating... The camera's actual image is overlaid on the digital doppelganger platform; and a spatial positioning module is communicatively connected to the digital doppelganger platform. When the spatial positioning module receives a push notification of an alarm point related to an event of interest to the user, the spatial positioning module notifies the digital doppelganger platform of an alarm, allowing the spatial positioning module to activate nearby available cameras based on the alarm point of the event of interest to obtain monitoring information of the event of interest to the user. Furthermore, the spatial positioning module provides or calculates a three-tiered list of recommended cameras based on the effectiveness of the cameras, including a first-tier list of low-effective cameras, a second-tier list of medium-effective cameras, and a third-tier list of high-effective cameras.

2. The video surveillance system as described in claim 1 further includes a spatial index database, wherein, The data linking module further calculates a spatial index structure based on the minimum bounding rectangle of the illuminated polygon range of the camera. The data linking module then uses this spatial index structure to insert information about the camera's description, position, orientation, and illuminated polygon range into the spatial index database for storage.

3. The video surveillance system as described in claim 1 further includes a spatial index database, wherein, The simulation rendering module further uses homography to calculate the UV mapping coordinates used for rendering in the digital avatar platform by using the data linking module to store the illumination polygon range of the camera and the connection of the camera in the spatial index database.

4. The video surveillance system as described in claim 1, wherein, The simulation rendering module further checks the update of the image stream of the projected camera during each rendering process. If the image stream of the camera has been updated, the simulation rendering module updates the image texture of the camera, but does not recalculate the UV mapping coordinates.

5. The video surveillance system as described in claim 1, wherein, The spatial positioning module further verifies the viewing distance and azimuth of the multiple cameras listed in the first-level low-efficiency camera recommendation list. The spatial positioning module calculates the multiple numbered cameras within a predetermined positive and negative degree range of the viewing distance and azimuth of the multiple cameras, so that the spatial positioning module can use the numbered list of the multiple numbered cameras as the second-level effective camera recommendation list and send it back to the digital clone platform.

6. The video surveillance system as described in claim 1, wherein, The spatial positioning module further performs spatial geometric calculations on the illumination polygon range of the multiple numbered cameras listed in the second-level effective camera suggestion list and the location information of the alarm point of the event of concern, so that the spatial positioning module can use the numbered cameras included in the result of the spatial geometric calculation as the third-level high-effective camera suggestion list and send it back to the digital clone platform.

7. The video surveillance system as described in claim 1, wherein, The spatial positioning module calculates the minimum bounding rectangle as the positioning range according to the camera coordinates of the specified low-efficiency camera suggestion list of the first level, the medium-efficiency camera suggestion list of the second level, or the high-efficiency camera suggestion list of the third level. The spatial positioning module then transmits the positioning range back to the digital clone platform.

8. The video surveillance system as described in claim 1, wherein, The spatial positioning module further uses a spatial indexing method to search for a list of low-efficiency cameras in the first level within a specified alarm range. Then, the spatial positioning module calculates based on the visible distance and azimuth of the cameras in the first level low-efficiency camera suggestion list to filter out the list of medium-efficiency cameras in the second level. Finally, the spatial positioning module performs spatial geometric calculations based on the illumination polygon range of the cameras in the second level medium-efficiency camera suggestion list to retain the list of high-efficiency cameras in the third level that meet the range conditions.

9. A method for image surveillance using a digital clone model, comprising: A data linking module creates a digital clone model of the camera based on the camera information. A simulation rendering module calculates a digital array of the camera's illumination polygon range based on the attributes of the digital clone model. Then, the simulation rendering module calculates UV mapping coordinates used for rendering in the digital clone platform based on the digital array of the camera's illumination polygon range, the camera's image width, and the camera's image height. The simulation rendering module then uses these UV mapping coordinates to create image textures from the camera's image stream and projects them onto the 3D model of the digital clone platform for overlay rendering, simulating the overlay effect of the camera's real image being presented on the digital clone platform. When a spatial positioning module receives a push notification about an alarm point related to an event of interest to the user, the spatial positioning module notifies the digital clone platform of an alarm. The spatial positioning module then activates nearby cameras based on the alarm point of the event of interest to obtain monitoring information about the event of interest to the user. The spatial positioning module provides or calculates a three-tiered list of recommended cameras based on their effectiveness: a first-tier list of low-effective cameras, a second-tier list of medium-effective cameras, and a third-tier list of high-effective cameras.

10. The image monitoring method as described in claim 9 further includes having the data link module calculate a spatial index structure based on the minimum bounding rectangle of the illumination polygon range of the camera, and having the data link module use the spatial index structure to insert information about the camera description, camera position, camera orientation and illumination polygon range of the camera into a spatial index database for storage.

11. The image monitoring method as described in claim 9 further includes using homography to calculate the UV mapping coordinates used for rendering in the digital avatar platform by means of the analog rendering module through the data linking module storing the illumination polygon range of the camera and the connection of the camera in a spatial index database.

12. The image monitoring method as described in claim 9 further includes, during each rendering, the simulation rendering module performing an update determination on the image stream of the camera that is turned on for projection; if the image stream of the camera has been updated, the simulation rendering module updates the image texture of the camera, but does not recalculate the UV mapping coordinates.

13. The image monitoring method as described in claim 9 further includes verifying the viewing distance and azimuth of the plurality of cameras listed in the low-efficiency camera suggestion list of the first level by the spatial positioning module, and calculating the plurality of numbered cameras within a predetermined positive and negative degree range of the viewing distance and azimuth of the plurality of cameras by the spatial positioning module, so that the spatial positioning module uses the number list of the plurality of numbered cameras as the effective camera suggestion list of the second level to be transmitted back to the digital clone platform.

14. The image surveillance method as described in claim 9 further includes the spatial positioning module performing spatial geometric calculations on the illumination polygon range of the multiple numbered cameras listed in the effective camera suggestion list of the second level and the location information of the alarm point of the event of concern, so that the spatial positioning module uses the numbered cameras included in the result of the spatial geometric calculation as the high-effectiveness camera suggestion list of the third level to be transmitted back to the digital clone platform.

15. The image monitoring method as described in claim 9 further includes having the spatial positioning module calculate the minimum bounding rectangle as the positioning range according to the camera coordinates of the specified first-level low-efficiency camera recommendation list, the second-level medium-efficiency camera recommendation list, or the third-level high-efficiency camera recommendation list, and then having the spatial positioning module transmit the positioning range back to the digital clone platform.

16. The image monitoring method as described in claim 9 further includes the spatial positioning module searching for a list of low-efficiency cameras in the first level within a specified alarm and a specified range using a spatial indexing method; the spatial positioning module then calculating based on the visible distance azimuth angle of the camera according to the list of low-efficiency cameras in the first level to filter out a list of medium-efficiency cameras in the second level; and the spatial positioning module then performing spatial geometric calculations based on the illumination polygon range of the camera according to the list of medium-efficiency cameras in the second level to retain a list of high-efficiency cameras in the third level that meet the range conditions.

17. A computer-readable medium, applied in a computing device or computer, storing instructions and executing the computer-readable medium via a processor and memory, so as to perform, when executing the computer-readable medium, the image monitoring method using a digital clone model as described in any one of claims 9 to 16.

Images