Monitoring and construction digital total plan dynamic and static fusion method and device and storage medium

Abstract

The invention relates to the technical field of monitoring and plain map dynamic and static fusion, and discloses a monitoring and construction digital total plain map dynamic and static fusion method and device, computing equipment and a storage medium, and the method comprises the steps: obtaining a construction digital total plain map; determining deployment points of video monitoring according to the construction digital general map; obtaining an original video stream shot by a monitoring camera; performing view perspective transformation operation on the original video stream to obtain an overlook view video stream; cutting and masking the overlook view angle video stream to obtain a to-be-spliced overlook view angle video stream matched with the construction digital total plain map; and splicing a to-be-spliced overlook view video stream on the construction digital general plain map, and fusing to generate a construction site monitoring dynamic and static fusion panorama. According to the method, the monitoring video is fused with the construction site general map through methods of perspective transformation, shielding, pasting and splicing and the like, the construction site dynamic panorama under the visual overall view angle is constructed, the digital supervision effect of a dynamic satellite map is achieved, and the construction management efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of dynamic and static fusion of monitoring video and plan, and more specifically to a method, device, computing equipment and storage medium for dynamic and static fusion of monitoring and construction digital master plan. Background Art

[0002] The construction site environment is complex, with a large flow of personnel and dense high-risk areas. The use of information technology to improve the efficiency of construction site control, reduce the probability of accidents, prevent the loss of important assets, and grasp the construction progress in real time is of great significance to the management of construction projects. At present, the construction site usually adopts three methods to inspect and monitor the construction site: manual inspection, drone inspection and fixed-point monitoring. The above three methods can achieve regular and fixed-point supervision of the construction site to a certain extent, but the disadvantages are also obvious, such as: high labor costs, high safety risks, inability to achieve real-time dynamic panoramic monitoring throughout the day, limited monitoring coverage area and fragmented images, it is difficult to quickly locate and combine the monitoring screen with the actual area, single-channel monitoring multiple operations and management difficulties, and difficulty in viewing multiple videos in different locations. It is beneficial to improve the efficiency of construction management by integrating scattered monitoring videos with the general plan of the construction site to construct a dynamic panoramic view of the construction site from an intuitive overall perspective. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, computing device and storage medium for the dynamic and static fusion of monitoring and construction digital master plan, which fuses the monitoring video with the construction site master plan through perspective transformation, cropping masking, mapping splicing and other methods, constructs a dynamic panoramic view of the construction site from an intuitive overall perspective, achieves a digital supervision effect similar to a dynamic satellite map, and improves construction management efficiency.

[0004] In a first aspect, an embodiment of the present application provides a method for integrating monitoring and construction digital master plan in both dynamic and static aspects, obtaining a construction digital master plan, which is a general layout of a construction site of a proposed project; determining deployment points of video monitoring of the construction site according to the construction digital master plan so that the video monitoring range covers the range of the construction digital master plan; after deploying the positions of multiple monitoring cameras according to the deployment points, obtaining original video streams captured by the multiple monitoring cameras; performing a perspective transformation operation on the original video stream to obtain a bird's-eye view video stream after the perspective transformation; performing cropping and masking processing on the bird's-eye view video stream to obtain a bird's-eye view video stream to be stitched that matches the construction digital master plan; stitching the bird's-eye view video stream to be stitched on the construction digital master plan, and fusing them to generate a dynamic and static fusion panoramic view of the construction site monitoring.

[0005] In one embodiment, performing a perspective transformation operation on the original video stream to obtain a top-view video stream after perspective transformation includes:

[0006] Transmitting the original video stream to a cloud server, wherein the cloud server includes a reading module and a perspective transformation module;

[0007] Reading the original video stream by the reading module to obtain original video source information;

[0008] Performing perspective transformation on the original video source information by the perspective transformation module to obtain perspective transformation information corresponding to the original video source information;

[0009] Sending the original video source information and the perspective transformation information to an edge server;

[0010] The edge server performs parsing operations on the original video source information and the perspective transformation information to obtain a bird's-eye view video stream after the perspective transformation of the original video stream.

[0011] In one embodiment, the original video source information includes multiple real-time frames, the perspective transformation information of the original video source information includes transformation matrix information when perspective transformation is performed on the multiple real-time frames, and the perspective transformation module performs perspective transformation on the original video source information to obtain the perspective transformation information corresponding to the original video source information, including:

[0012] Taking one of the real-time frames as the target real-time frame, extracting the target feature points of the target real-time frame using the feature point extraction algorithm in the perspective view transformation module, and obtaining the original coordinates of the target feature points, wherein each of the target real-time frames corresponds to a target top-down view picture after perspective transformation;

[0013] Determine the target coordinates of the target feature point in the target top-down perspective image;

[0014] Calculate a target transformation matrix for perspective transformation of the target real-time frame according to the original coordinates and the target coordinates of the target feature points to obtain target transformation matrix information;

[0015] After obtaining the target transformation matrix information of the perspective transformation of each target real-time frame, the viewing angle perspective transformation information of the original video source information is obtained.

[0016] In one embodiment, the target transformation matrix for perspective transformation of the target real-time frame is calculated according to the original coordinates and the target coordinates of the target feature points to obtain the target transformation matrix information, including:

[0017] The original coordinates and the target coordinates of the target feature points are substituted into a preset perspective transformation formula to solve the formula and obtain target transformation matrix information.

[0018] In one embodiment, the original video source information includes multiple real-time frames, the perspective transformation information of the original video source information includes transformation matrix information when the perspective transformation is performed on the multiple real-time frames, and the edge server performs a parsing operation on the original video source information and the perspective transformation information to obtain a bird's-eye view video stream after the perspective transformation of the original video stream, including:

[0019] Taking a real-time frame as a target real-time frame, the edge server calculates the top-view coordinates of all pixel coordinates of the target real-time frame in the target top-view image according to the target transformation matrix information corresponding to the target real-time frame, wherein each target real-time frame corresponds to a target top-view image after perspective transformation;

[0020] Generate a target overlooking perspective image according to the overlooking perspective coordinates of the target real-time frame and the target real-time frame;

[0021] After completing the generation of the target bird's-eye view picture of each target real-time frame, all the target bird's-eye view pictures are encoded to obtain a bird's-eye view video stream after the original video stream has been transformed by perspective.

[0022] In one embodiment, the step of performing cropping and masking processing on the bird's-eye view video stream to obtain the bird's-eye view video stream to be stitched that matches the construction digital master plan includes:

[0023] Simulating and splicing the multiple bird's-eye view video streams on the construction digital master plan;

[0024] Marking a repeated area formed by simulating and stitching the multiple top-view video streams as a mask area;

[0025] The masked area is cropped and masked using video mask technology to obtain multiple bird's-eye view video streams to be stitched together that match the construction digital master plan, wherein when the multiple bird's-eye view video streams to be stitched together are completely stitched together on the construction digital master plan, there are no repeated areas between any two bird's-eye view video streams to be stitched together.

[0026] In one embodiment, after the bird's-eye view video stream to be stitched is stitched on the digital construction master plan to generate a dynamic and static fusion panoramic view of the construction site monitoring, the method further includes:

[0027] Uploading the dynamic and static fusion panoramic image to a cloud server, and generating an H5 webpage link according to the dynamic and static fusion panoramic image;

[0028] If a request for browsing the H5 web page link is obtained from a terminal device, the dynamic and static fusion panoramic image is transmitted to the edge server in a P2P communication manner through the cloud server, and the edge server forwards the H5 web page link to the terminal device.

[0029] In a second aspect, an embodiment of the present application provides a device for the dynamic and static fusion of a monitoring and construction digital master plan, which has the function of implementing the dynamic and static fusion method of the monitoring and construction digital master plan provided in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware. In one embodiment, the device for the dynamic and static fusion of the monitoring and construction digital master plan includes:

[0030] The first acquisition module is used to acquire a digital construction master plan, which is a general layout of the construction site of the proposed project;

[0031] A determination module is used to determine the deployment points of the video surveillance of the construction site according to the digital construction master plan so that the video surveillance range covers the range of the digital construction master plan;

[0032] A second acquisition module: configured to acquire original video streams captured by the plurality of surveillance cameras after the plurality of surveillance cameras are deployed according to the deployment points;

[0033] Transformation module: used for performing perspective transformation operation on the original video stream to obtain a bird's-eye view video stream after perspective transformation;

[0034] Mask module: used for performing cropping and masking processing on the bird's-eye view video stream to obtain a bird's-eye view video stream to be spliced ​​that matches the construction digital master plan;

[0035] Fusion module: The bird's-eye view video stream to be stitched is stitched on the digital construction master plan to generate a dynamic and static fusion panoramic view of the construction site monitoring.

[0036] In one embodiment, the transformation module is specifically used to:

[0037] Transmitting the original video stream to a cloud server, wherein the cloud server includes a reading module and a perspective transformation module;

[0038] Reading the original video stream by the reading module to obtain original video source information;

[0039] Performing perspective transformation on the original video source information by the perspective transformation module to obtain perspective transformation information corresponding to the original video source information;

[0040] Sending the original video source information and the perspective transformation information to an edge server;

[0041] The edge server performs parsing operations on the original video source information and the perspective transformation information to obtain a bird's-eye view video stream after the perspective transformation of the original video stream.

[0042] In one embodiment, the original video source information includes multiple real-time frames, the perspective transformation information of the original video source information includes transformation matrix information when perspective transformation is performed on the multiple real-time frames, and the transformation module is further used to:

[0043] Taking one of the real-time frames as the target real-time frame, extracting the target feature points of the target real-time frame using the feature point extraction algorithm in the perspective view transformation module, and obtaining the original coordinates of the target feature points, wherein each of the target real-time frames corresponds to a target top-down view picture after perspective transformation;

[0044] Determine the target coordinates of the target feature point in the target top-down perspective image;

[0045] Calculate a target transformation matrix for perspective transformation of the target real-time frame according to the original coordinates and the target coordinates of the target feature points to obtain target transformation matrix information;

[0046] After obtaining the target transformation matrix information of the perspective transformation of each target real-time frame, the viewing angle perspective transformation information of the original video source information is obtained.

[0047] In one embodiment, the transformation module is further configured to:

[0048] The original coordinates and the target coordinates of the target feature points are substituted into a preset perspective transformation formula to solve the formula and obtain target transformation matrix information.

[0049] In one embodiment, the original video source information includes multiple real-time frames, the perspective transformation information of the original video source information includes transformation matrix information when perspective transformation is performed on the multiple real-time frames, and the transformation module is further used to:

[0050] Taking a real-time frame as a target real-time frame, the edge server calculates the top-view coordinates of all pixel coordinates of the target real-time frame in the target top-view image according to the target transformation matrix information corresponding to the target real-time frame, wherein each target real-time frame corresponds to a target top-view image after perspective transformation;

[0051] Generate a target overlooking perspective image according to the overlooking perspective coordinates of the target real-time frame and the target real-time frame;

[0052] After completing the generation of the target bird's-eye view picture of each target real-time frame, all the target bird's-eye view pictures are encoded to obtain a bird's-eye view video stream after the original video stream has been transformed by perspective.

[0053] In one embodiment, the mask module is specifically used for:

[0054] Simulating and splicing the multiple bird's-eye view video streams on the construction digital master plan;

[0055] Marking a repeated area formed by simulating and stitching the multiple top-view video streams as a mask area;

[0056] The masked area is cropped and masked using video mask technology to obtain multiple bird's-eye view video streams to be stitched together that match the construction digital master plan, wherein when the multiple bird's-eye view video streams to be stitched together are completely stitched together on the construction digital master plan, there are no repeated areas between any two bird's-eye view video streams to be stitched together.

[0057] In one embodiment, the monitoring and construction digital master plan dynamic and static fusion device is further used for:

[0058] Uploading the dynamic and static fusion panoramic image to a cloud server, and generating an H5 webpage link according to the dynamic and static fusion panoramic image;

[0059] If a request for browsing the H5 web page link is obtained from a terminal device, the dynamic and static fusion panoramic image is transmitted to the edge server in a P2P communication manner through the cloud server, and the edge server forwards the H5 web page link to the terminal device.

[0060] In a third aspect, an embodiment of the present application provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for dynamic and static fusion of monitoring and construction digital master plans as described in the first aspect when executing the computer program.

[0061] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions, which, when executed on a computer, enables the computer to execute the method for dynamic and static fusion of monitoring and construction digital master plan as described in the first aspect.

[0062] This application combines the surveillance video with the construction site master plan through perspective transformation, cropping and masking, and map stitching to construct a dynamic panoramic view of the construction site from an intuitive overall perspective, achieve a dynamic satellite map-like digital supervision effect, and improve construction management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0064] Figure 1 It is a scene schematic diagram of a dynamic and static fusion device for monitoring and construction digital master plan provided in an embodiment of the present application;

[0065] Figure 2 It is a schematic flow chart of an embodiment of the method for integrating dynamic and static monitoring and construction digital master plan provided in the embodiments of the present application;

[0066] Figure 3 A schematic diagram of a scenario of an embodiment of the method for integrating dynamic and static monitoring and construction digital master plan in an embodiment of the present application;

[0067] Figure 4 It is a schematic diagram of a scene of an embodiment of mask cutting of the method for dynamic and static fusion of monitoring and construction digital master plan provided in the embodiment of the present application;

[0068] Figure 5 This is a schematic diagram of an example scenario of forwarding traffic of the method for integrating dynamic and static monitoring and construction digital master plan provided in the embodiment of the present application.

[0069] Figure 6 This is a schematic structural diagram of a method and device for integrating dynamic and static monitoring and construction digital master plan according to an embodiment of the present application;

[0070] Figure 7 A schematic diagram of the structure of a dynamic and static fusion device for monitoring and construction digital master plan according to an embodiment of the present application;

[0071] Figure 8 A schematic diagram of the structure of a mobile phone in an embodiment of the present application;

[0072] Fig. 9 A schematic diagram of the structure of a server in an embodiment of the present application.

[0073] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0074] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0075] In the following description, the specific embodiments of the present application will be described with reference to the steps and symbols performed by one or more computers, unless otherwise specified. Therefore, these steps and operations will be mentioned several times as being performed by a computer, and the computer execution referred to herein includes the operation of a computer processing unit by an electronic signal representing data in a structured form. This operation converts the data or maintains it at a location in the memory system of the computer, which can be reconfigured or otherwise change the operation of the computer in a manner familiar to testers in the field. The data structure maintained by the data is a physical location in the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above text, which does not represent a limitation, and testers in the field will understand that the various steps and operations described below can also be implemented in hardware.

[0076] The terms "module" or "unit" used herein may be considered as software objects executed on the computing system. The different components, modules, engines and services described herein may be considered as implementation objects on the computing system. The devices and methods described herein are preferably implemented in software, but may also be implemented in hardware, all within the scope of protection of this application.

[0077] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0078] The embodiments of the present application provide a method, device and storage medium for integrating dynamic and static aspects of monitoring and construction digital master plan.

[0079] See also Figure 1 , Figure 1The present invention is a scene diagram of a dynamic and static fusion device for monitoring and construction digital master plan provided in an embodiment of the present application. The dynamic and static fusion device for monitoring and construction digital master plan may include a dynamic and static fusion system 100 for monitoring and construction digital master plan and a user terminal 200. The dynamic and static fusion system 100 for monitoring and construction digital master plan is connected via a network, and the dynamic and static fusion device for monitoring and construction digital master plan is integrated in the dynamic and static fusion system 100. In the embodiment of the present application, the dynamic and static fusion system 100 for monitoring and construction digital master plan may be a terminal device or a server, and the dynamic and static fusion system 100 for monitoring and construction digital master plan may send a dynamic and static fusion panoramic image to the user terminal 200.

[0080] In the embodiment of the present application, when the dynamic and static fusion system 100 of the monitoring and construction digital master plan is a server, the server can be an independent server, or a server network or server cluster composed of servers. For example, the server described in the embodiment of the present application includes but is not limited to a computer, a network host, a single network server, a plurality of network server sets or a cloud server composed of a plurality of servers. Among them, the cloud server is composed of a large number of computers or network servers based on cloud computing (Cloud Computing). In the embodiment of the present application, communication between the server and the client can be achieved through any communication method, including but not limited to mobile communications based on the 3rd Generation Partnership Project (3GPP), Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WiMAX), or computer network communications based on TCP / IP Protocol suite (TCP / IP), User Datagram Protocol (UDP) protocol, etc.

[0081] It is understandable that when the monitoring and construction digital master plan dynamic and static fusion system 100 used in the embodiment of the present application is a terminal device, the terminal device can be a device that includes both receiving hardware and transmitting hardware, that is, a device with receiving and transmitting hardware that can perform two-way communication on a two-way communication link. Such terminal devices may include: cellular or other communication devices, which have a single-line display or a multi-line display or a cellular or other communication device without a multi-line display. The specific monitoring and construction digital master plan dynamic and static fusion system 100 can be a desktop terminal or a mobile terminal. The monitoring and construction digital master plan dynamic and static fusion system 100 can be one of a mobile phone, a tablet computer, a laptop computer, etc.

[0082] The terminal device involved in the embodiments of the present application may also refer to a device that provides voice and / or data connectivity to a user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. For example, a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device, which exchanges voice and / or data with a wireless access network. For example, personal communication service (English full name: Personal Communication service, English abbreviation: PCs) phone, cordless phone, session initiation protocol (sIP) phone, wireless local loop (Wireless Local Loop, English abbreviation: WLL) station, personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA) and other devices.

[0083] Those skilled in the art will understand that Figure 1 The application environment shown in the figure is only one application scenario of the present application solution and does not constitute a limitation on the application scenario of the present application solution. Other application environments may also include Figure 1 More or fewer computing devices, or computing device network connection relationships, such as Figure 1 Only one computing device is shown. It can be understood that the monitoring and construction digital master plan dynamic and static fusion device can also include one or more other computing devices, or / and one or more other computing devices connected to the monitoring and construction digital master plan dynamic and static fusion system 100 network, which is not specifically limited here.

[0084] In addition, if Figure 1 As shown, the monitoring and construction digital master plan dynamic and static fusion device can also include a memory 300 for storing data, such as storing original video streams, bird's-eye view video streams, etc.

[0085] It should be noted that Figure 1 The scene diagram of the device for dynamic and static fusion of monitoring and construction digital master plan shown is merely an example. The device for dynamic and static fusion of monitoring and construction digital master plan and the scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the device for dynamic and static fusion of monitoring and construction digital master plan and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0086] This application integrates the monitoring video with the construction site master plan through perspective transformation, cropping mask, map splicing and other methods to construct a dynamic panoramic view of the construction site from an intuitive overall perspective, achieve a digital supervision effect similar to a dynamic satellite map, and improve the efficiency of construction management. The following is a detailed description in conjunction with specific embodiments.

[0087] In this embodiment, the method for integrating the static and dynamic aspects of the monitoring and construction digital master plan will be described. The method for integrating the static and dynamic aspects of the monitoring and construction digital master plan can be specifically integrated into the system 100 for integrating the static and dynamic aspects of the monitoring and construction digital master plan.

[0088] The present application provides a method for the dynamic and static fusion of monitoring and a digital construction plan, and the method comprises: obtaining a digital construction plan, which is a general layout of a construction site of a proposed project; determining deployment points of video surveillance of the construction site according to the digital construction plan so that the video surveillance range covers the range of the digital construction plan; after deploying the positions of a plurality of surveillance cameras according to the deployment points, obtaining original video streams shot by the plurality of surveillance cameras; performing a perspective transformation operation on the original video stream to obtain a bird's-eye view video stream after the perspective transformation; performing a cropping and masking process on the bird's-eye view video stream to obtain a bird's-eye view video stream to be spliced ​​that matches the digital construction plan; splicing the bird's-eye view video stream to be spliced ​​on the digital construction plan, and fusing to generate a dynamic and static fusion panoramic picture of the construction site monitoring.

[0089] See also Figure 2 and Figure 3 , Figure 2 This is a flow chart of an embodiment of the method for integrating dynamic and static monitoring and construction digital master plan in the embodiment of the present application. Figure 3 The following is a schematic diagram of an embodiment of the method for integrating dynamic and static monitoring and construction digital master plan in the embodiment of the present application. The method for integrating dynamic and static monitoring and construction digital master plan includes the following steps 201 to 206:

[0090] 201. Obtain the digital construction master plan.

[0091] Specifically, the digital construction master plan is the general layout of the construction site of the proposed project. Optionally, the digital construction master plan can indicate the location and size of all above-ground and proposed buildings and other facilities, and can also indicate all temporary buildings and roads built for construction. Figure 1Generally, it is drawn before the construction of the proposed project begins according to the construction plan and the existing buildings on the construction site. Optionally, the digital construction master plan can also be modified and updated accordingly as the construction site changes during the construction of the proposed project. Obtaining the latest version of the digital construction master plan and making a dynamic and static fusion panoramic view of the master plan can improve the efficiency of construction management.

[0092] 202. Determine deployment points of video surveillance of the construction site according to the digital construction master plan so that a video surveillance range covers a range of the digital construction master plan.

[0093] Specifically, before the dynamic and static fusion of the surveillance video and the digital master plan, it is necessary to ensure that the surveillance range of the surveillance video of the construction site can cover the entire construction site, that is, the surveillance range of the surveillance video must cover the range of the digital master plan. Optionally, after obtaining the digital master plan, plan the layout of the video surveillance based on the digital master plan, determine the deployment points and coverage areas of each surveillance video, and after ensuring that the surveillance range of the surveillance video can cover the digital master plan, deploy each surveillance camera according to the layout, record the angle of the camera, ensure that the area to be cropped and spliced ​​corresponding to the digital master plan is within the effective shooting range of the surveillance camera, make good annotations for the video shot by each surveillance camera, and regularly check the shooting angle of the surveillance camera to ensure that the surveillance video shot by the surveillance camera corresponds to the deployment plan of the digital master plan, so as to facilitate the subsequent dynamic and static fusion of the video stream shot by the surveillance camera.

[0094] 203. After the positions of the multiple surveillance cameras are deployed according to the deployment points, original video streams captured by the multiple surveillance cameras are obtained.

[0095] Specifically, the operation of perspective transformation of the original video stream captured by the surveillance camera can be completed by the edge server and the cloud server in collaboration. After the surveillance cameras are deployed according to the layout of the digital master plan, the video stream of each corresponding surveillance camera is connected to the video management service of the cloud server. Optionally, the cloud server includes a reading module, through which the original video stream can be previewed and read to obtain video source information. Optionally, the video source information can be each real-time frame of the original video stream. Optionally, the cloud server includes a perspective transformation module. When the original video stream is perspective transformed by the cloud server, the operation information of the perspective transformation of each single-channel video stream can be recorded, and the perspective perspective transformation information obtained when the original video stream is perspective transformed can be obtained. Optionally, the perspective perspective transformation information can include perspective transformation matrix information.

[0096] 204. Perform a perspective transformation operation on the original video stream to obtain a bird's-eye view video stream after the perspective transformation.

[0097] Specifically, perspective transformation is to project a two-dimensional image onto a three-dimensional viewing plane, and then convert it to two-dimensional coordinates, so it is also called projection mapping, which is a process of converting two-dimensional to three-dimensional and then to two-dimensional.

[0098] Optionally, the original video stream is read by the reading module of the cloud server to obtain the original video source information, and the original video source information may include the real-time frame of the original video stream. It is understandable that a video stream is composed of multiple real-time frames. Optionally, the original video source information is transformed by the perspective viewpoint transformation module of the cloud server to obtain the perspective viewpoint transformation information corresponding to the original video source information, and the perspective viewpoint transformation information may include perspective transformation matrix information. Specifically, each real-time frame corresponds to a perspective transformation matrix. The new coordinates obtained by performing matrix operations on the pixel coordinates of the real-time frame and the perspective transformation matrix can obtain the real-time frame after perspective transformation, that is, the real-time frame of the top-down perspective, that is, the target real-time frame after perspective transformation. Optionally, the original video source information and the perspective viewpoint transformation information are sent to the edge server; the original video source information and the perspective viewpoint transformation information are parsed by the edge server to obtain the top-down perspective video stream after the original video stream is transformed by the perspective viewpoint. Optionally, when the cloud server performs a perspective transformation on the real-time frames of the original video stream, the operation information, i.e., the corresponding video source information and the perspective transformation information, is recorded, and the video source information and the perspective transformation information are packaged and sent to the edge server. After obtaining the packaged information, the edge server parses the packaged information and uses the video processing service in the edge server to decode the original video stream, obtain the real-time frame image of the original video stream through the video source information analysis, and then perform a coordinate projection transformation on the real-time frame image according to the perspective transformation information, i.e., the perspective transformation matrix, and uniformly transform the perspective of each real-time frame image into a bird's-eye view. Then, each real-time frame after the perspective transformation is re-encoded in sequence to generate a new video stream, i.e., a bird's-eye view video stream, that is, a new webtrc video stream is generated through the edge server real-time communication technology (Web Real-Time Communications, webtrc), and then the bird's-eye view video stream is sent to the cloud server, thereby completing the perspective transformation operation of the single-channel surveillance video.

[0099] 205. Perform cropping and masking processing on the bird's-eye view video stream to obtain a bird's-eye view video stream to be stitched that matches the construction digital master plan.

[0100] Specifically, multiple bird's-eye view video streams are simulated and stitched together on the digital construction master plan; the repeated area formed after the simulated stitching of the multiple bird's-eye view video streams is marked as a mask area; the mask area is cropped and masked using video mask masking technology to obtain multiple bird's-eye view video streams to be stitched that match the digital construction master plan.

[0101] Optionally, the video cropping and masking operation can be completed on the cloud server. After the video perspective transformation is completed, each monitored video is converted to a bird's-eye view, which is consistent with the view of the digital master plan. However, there are certain repeated areas in the video coverage of the surveillance camera. If there are repeated areas between different video streams, it is difficult to connect the pictures between the videos, and it is impossible to achieve complete consistency when splicing and merging with the digital master plan. Therefore, it is understandable that multiple bird's-eye view video streams are simulated spliced ​​on the construction digital master plan. During the splicing process, the repeated areas formed after the splicing of multiple video streams are marked as mask areas. It is understandable that when there are repeated areas between two video streams, the repeated area of ​​one of the video streams can be marked as a mask area. The video mask masking technology is used to mask multiple video streams according to the digital master plan, that is, the repeated area between any two video streams, that is, the mask area, is masked with a transparent color, so that multiple video streams are completely consistent when spliced ​​on the digital master plan. The multiple bird's-eye view video streams after masking are the bird's-eye view video streams to be spliced. It can be understood that when multiple bird's-eye view video streams to be stitched together are completely stitched together on the digital master plan of construction, there are no overlapping areas between any two bird's-eye view video streams to be stitched together, which facilitates subsequent monitoring and dynamic and static fusion of the digital master plan.

[0102] 206. The bird's-eye view video stream to be stitched is stitched on the digital construction master plan to generate a dynamic and static fusion panoramic view of the construction site monitoring.

[0103] Specifically, the dynamic and static fusion of the bird's-eye view video stream to be stitched and the digital master plan of the construction can be completed on the cloud server. The masked video stream, that is, the bird's-eye view video stream to be stitched, is stitched and pasted on the digital master plan of the construction according to the pre-planned area and the corresponding area. Optionally, when collaging video streams on the cloud server, the effect of the dynamic and static fusion of the video stream and the digital master plan of the construction can be previewed at any time. If the fusion effect is not good or the perspective does not meet the preset requirements, the perspective transformation and cropping mask processing of the single-channel video stream can be re-performed until the fusion effect meets the preset requirements. Optionally, after the fusion generates the dynamic and static fusion panoramic view of the construction site monitoring, a corresponding H5 web page link can also be generated. When receiving the browsing request sent by the user, the H5 web page link can be sent to the terminal device held by the user. The user can view the panoramic monitoring screen of the construction site in real time through terminal devices such as mobile phones, tablets, and computers.

[0104] The embodiment of the present application combines the surveillance video with the construction site master plan through perspective transformation, cropping and masking, and map splicing to construct a dynamic panoramic view of the construction site from an intuitive overall perspective, achieve a dynamic satellite map-like digital supervision effect, and improve construction management efficiency.

[0105] In one implementation of the present application, performing a perspective transformation operation on an original video stream to obtain a top-view video stream after perspective transformation includes:

[0106] The original video stream is transmitted to a cloud server, wherein the cloud server includes a reading module and a perspective transformation module; the original video stream is read by the reading module to obtain the original video source information; the perspective transformation module performs perspective transformation on the original video source information to obtain perspective transformation information corresponding to the original video source information; the original video source information and the perspective transformation information are sent to an edge server; the edge server performs a parsing operation on the original video source information and the perspective transformation information to obtain a bird's-eye view video stream after the original video stream has been transformed by perspective transformation.

[0107] Specifically, the original video stream is read by the reading module of the cloud server to obtain the original video source information, and the original video source information may include the real-time frame of the original video stream. It can be understood that a video stream is composed of multiple real-time frames. Optionally, the original video source information is subjected to a perspective perspective transformation by the perspective perspective transformation module of the cloud server to obtain the perspective perspective transformation information corresponding to the original video source information, and the perspective perspective transformation information may include perspective transformation matrix information. Specifically, each real-time frame corresponds to a perspective transformation matrix, and the new coordinates obtained by performing matrix operations on the pixel coordinates of the real-time frame and the perspective transformation matrix can obtain the real-time frame after perspective transformation, that is, the real-time frame of the top-down perspective, that is, the target real-time frame after perspective transformation. Optionally, the original video source information and the perspective perspective transformation information are sent to the edge server; the original video source information and the perspective perspective transformation information are analyzed by the edge server to obtain the top-down perspective video stream after the original video stream is transformed by the perspective perspective. Optionally, when the cloud server performs a perspective transformation on the real-time frames of the original video stream, the operation information, i.e., the corresponding video source information and the perspective transformation information, is recorded, and the video source information and the perspective transformation information are packaged and sent to the edge server. After obtaining the packaged information, the edge server parses the packaged information and uses the video processing service in the edge server to decode the original video stream, obtain the real-time frame image of the original video stream through the video source information analysis, and then perform a coordinate projection transformation on the real-time frame image according to the perspective transformation information, i.e., the perspective transformation matrix, and uniformly transform the perspective of each real-time frame image into a bird's-eye view. Then, each real-time frame after the perspective transformation is re-encoded in sequence to generate a new video stream, i.e., a bird's-eye view video stream, that is, a new webtrc video stream is generated through the edge server real-time communication technology (Web Real-Time Communications, webtrc), and then the bird's-eye view video stream is sent to the cloud server, thereby completing the perspective transformation operation of the single-channel surveillance video.

[0108] The embodiment of the present application uses cloud services and edge services as the basic technical foundation to complete the perspective conversion of the surveillance video, facilitating the subsequent dynamic and static fusion.

[0109] In one implementation of the present application, the original video source information includes a plurality of real-time frames, and the perspective transformation information of the original video source information includes transformation matrix information when the perspective transformation is performed on the plurality of real-time frames. The perspective transformation module performs perspective transformation on the original video source information to obtain the perspective transformation information corresponding to the original video source information, including:

[0110] Taking one of the real-time frames as the target real-time frame, the target feature points of the target real-time frame are extracted using the feature point extraction algorithm in the perspective viewpoint transformation module, and the original coordinates of the target feature points are obtained, wherein each target real-time frame corresponds to a target overhead view picture after perspective transformation; the target coordinates of the target feature points in the target overhead view picture are determined; the target transformation matrix for perspective transformation of the target real-time frame is calculated according to the original coordinates and target coordinates of the target feature points, and the target transformation matrix information is obtained; after obtaining the target transformation matrix information for perspective transformation of each target real-time frame, the perspective perspective transformation information of the original video source information is obtained.

[0111] Optionally, when one of the real-time frames is used as the target real-time frame, the feature points of the target real-time frame can be extracted by using a preset feature point extraction algorithm. Optionally, at least 4 feature points are extracted and the original coordinates of the feature points on the target real-time frame are obtained. It can be understood that each original video stream is composed of multiple real-time frames. When the perspective transformation is performed, each real-time frame will correspond to a perspective-transformed top-down perspective picture. Optionally, the target coordinates of the target feature points after the perspective transformation can be determined according to the position of the feature points on the top-down perspective picture. The preset algorithm can be used to automatically calculate the coordinates of the target feature points after the transformation to the top-down perspective, which is not limited here. After obtaining the original coordinates and target coordinates of at least 4 feature points, these 8 coordinates are input into the preset perspective transformation formula, and the perspective transformation matrix used when completing the perspective transformation can be solved. After calculating the perspective transformation matrix used when each target real-time frame is perspectively transformed, the top-down perspective picture after the perspective transformation can be obtained through the target real-time frame and the perspective transformation matrix.

[0112] The embodiment of the present application calculates the perspective transformation matrix of the viewing angle through coordinate operation, so as to facilitate the subsequent perspective transformation of the original video stream to achieve dynamic and static fusion.

[0113] In one embodiment of the present application, a target transformation matrix for perspective transformation of the target real-time frame is calculated according to the original coordinates and the target coordinates of the target feature points to obtain target transformation matrix information, including:

[0114] The original coordinates and the target coordinates of the target feature points are substituted into a preset perspective transformation formula to solve the formula and obtain target transformation matrix information.

[0115] Specifically, the expression of the preset perspective transformation matrix is ​​as follows:

[0116]

[0117] Among them, (Z,Y,Z) is the original plane coordinate point of the real-time frame, and the corresponding image plane coordinate point after perspective transformation is (X′,Y′,Z′). represents a linear transformation, [a 31 ,a 32 ] represents perspective transformation, and the others represent translation transformation. In the process of converting 3D to 2D, the 3D coordinates are converted to 2D coordinates by dividing by the z axis, that is, setting Z′=1, and dividing the transformed image coordinates by Z′, thereby converting from 3D coordinates to 2D coordinates. Specifically, a 33 =1, the following equation can be obtained by expanding the formula:

[0118] Z=a 31 x+a 32 y+a 33

[0119]

[0120] There are 8 unknowns, so 4 points on the original image and 4 points on the target image are needed to form 8 equations to solve the transformation matrix. It can be understood that obtaining the coordinates of at least 4 feature points when extracting feature points is also for solving the transformation matrix.

[0121] Specifically, when the 8 coordinates are (x0, y0), (x1, y1), (x2, y2), (x3, y3) and (x′0, y′0), (x′1, y′1), (x′2, y′2), (x′3, y′3), the equation can be obtained:

[0122]

[0123] The corresponding matrix form equation is:

[0124]

[0125] Specifically, after solving the corresponding equation, the perspective transformation matrix corresponding to the perspective transformation of the target real-time frame can be obtained. Optionally, the process of solving the transformation matrix can also be combined with the process of calculating the perspective transformed top-down view picture according to the transformation matrix in one algorithm. The perspective transformed top-down view picture can be directly obtained from the real-time frame through the preset algorithm, so as to obtain the perspective transformed top-down view video stream through encoding.

[0126] The embodiment of the present application solves the transformation matrix through a formula, which facilitates the subsequent perspective viewpoint transformation of the original video stream.

[0127] In one implementation of the present application, the original video source information includes multiple real-time frames, and the perspective transformation information of the original video source information includes transformation matrix information when the perspective transformation is performed on the multiple real-time frames. The edge server performs a parsing operation on the original video source information and the perspective transformation information to obtain a bird's-eye view video stream after the perspective transformation of the original video stream, including:

[0128] A real-time frame is taken as a target real-time frame, and the edge server calculates the overhead perspective coordinates of all pixel coordinates of the target real-time frame in a target overhead perspective picture according to the target transformation matrix information corresponding to the target real-time frame, wherein each target real-time frame corresponds to a target overhead perspective picture after perspective transformation; a target overhead perspective picture is generated according to the overhead perspective coordinates of the target real-time frame and the target real-time frame; after completing the generation of the target overhead perspective picture of each target real-time frame, all target overhead perspective pictures are encoded to obtain an overhead perspective video stream after the original video stream has been transformed through perspective perspective transformation.

[0129] Optionally, the cloud server packages the video source information and the perspective transformation information and sends them to the edge server. After obtaining the packaged information, the edge server parses the packaged information and uses the video processing service in the edge server to decode the original video stream, obtain the real-time frame image of the original video stream through the video source information analysis, and then perform coordinate projection transformation on the real-time frame image according to the perspective transformation information, that is, the perspective transformation matrix, and uniformly transform the perspective of each real-time frame image into a bird's-eye view. Specifically, the bird's-eye view coordinates of each pixel point on the real-time frame on the corresponding bird's-eye view image can be calculated through the pixel coordinates of the real-time frame and the corresponding target transformation matrix. After obtaining the coordinate information corresponding to each pixel point, the perspective transformation of a single real-time frame can be completed, and the bird's-eye view image after the perspective transformation of the single real-time frame can be obtained. After obtaining the bird's-eye view image after the perspective transformation of each real-time frame, each real-time frame after the perspective transformation is re-encoded in sequence to generate a new video stream, namely, the bird's-eye view video stream. That is, a new webtrc video stream is generated through the edge server webtrc (Web Real-Time Communications, real-time communication technology) service, and then the bird's-eye view video stream is sent to the cloud server, thus completing the perspective transformation operation of the single-channel monitoring video.

[0130] The embodiment of the present application performs perspective transformation processing on real-time monitoring of the construction site from different perspectives, so that multiple monitoring perspectives are uniformly transformed into a bird's-eye view, which facilitates the splicing of multiple monitoring video images and reduces the sense of fragmentation and spatial dislocation of different monitoring video images.

[0131] In one implementation of the present application, a bird's-eye view video stream is subjected to cropping and masking processing to obtain a bird's-eye view video stream to be stitched that matches the digital master plan of the construction, including:

[0132] Simulate and stitch multiple bird's-eye view video streams on a digital construction master plan; mark a repeated area formed after the simulated stitching of the multiple bird's-eye view video streams as a mask area; and use a video masking technique to crop and mask the mask area to obtain multiple bird's-eye view video streams to be stitched that match the digital construction master plan, wherein when the multiple bird's-eye view video streams to be stitched are completely stitched on the digital construction master plan, there is no repeated area between any two bird's-eye view video streams to be stitched.

[0133] Optionally, the video cropping and masking operations can be completed on the cloud server. After the perspective transformation of the video view is completed, each monitoring video is converted into a bird's-eye view, which is consistent with the view of the digital master plan. However, there are certain overlapping areas in the video coverage of the surveillance camera. If there are overlapping areas between different video streams, it is difficult to connect the images between the videos, and it is impossible to achieve complete consistency when splicing and fusing with the digital master plan. Therefore, it is understandable that multiple bird's-eye view video streams are simulated and spliced ​​on the construction digital master plan. During the splicing process, the overlapping areas formed after the splicing of multiple video streams are marked as mask areas. Figure 4 As shown, Figure 4 This is a schematic diagram of an embodiment scene of mask cropping in the embodiment of the present application. In the simulation splicing process, the mask area is masked with a transparent color, so that multiple pictures are completely matched during the splicing process. It is understandable that when there are repeated areas in two video streams, the repeated area of ​​one of the video streams can be marked as a mask area. The video mask masking technology is used to mask multiple video streams according to the digital master plan, that is, the repeated area between any two video streams, that is, the mask area, is masked with a transparent color, so that the multiple video streams are completely matched when spliced ​​on the digital master plan, and the multiple bird's-eye view video streams after the masking process are the bird's-eye view video streams to be spliced. It is understandable that when multiple bird's-eye view video streams to be spliced ​​are completely spliced ​​on the construction digital master plan, there is no repeated area between any two bird's-eye view video streams to be spliced, which is convenient for subsequent monitoring and dynamic and static fusion of the digital master plan.

[0134] The embodiment of the present application performs cropping and masking processing on video streams with different overhead perspectives, removes the overlapping areas of multiple surveillance videos, increases the connectivity of adjacent video stitching images, achieves matching stitching of multiple surveillance videos, and improves the overall sense of the image.

[0135] In one embodiment of the present application, after the bird's-eye view video stream to be stitched is stitched on the digital master plan of the construction site to generate a dynamic and static fusion panoramic view of the construction site monitoring, the method for dynamic and static fusion of monitoring and the digital master plan of the construction site further includes:

[0136] The dynamic and static fusion panoramic image is uploaded to the cloud server, and an H5 web link is generated based on the dynamic and static fusion panoramic image; if a request for browsing the H5 web link is received from a terminal device, the dynamic and static fusion panoramic image is transmitted to the edge server via P2P communication through the cloud server, and the edge server forwards the H5 web link to the terminal device.

[0137] Optional, such as Figure 5 As shown, Figure 5 This is a schematic diagram of an example scenario of forwarding traffic in the embodiment of the present application. After the generated dynamic and static fusion panoramic image is published to the cloud server, when the user plays the panoramic monitoring screen in real time through the browser on the mobile phone, tablet, computer and other devices, the dynamic and static fusion video stream, that is, the H5 web page link can be transmitted in the form of P2P communication, that is, the video stream does not pass through the cloud, and the traffic is directly forwarded by the edge server, which can realize cross-segment, point-to-point two-way information transmission, greatly reducing the bandwidth pressure of uploading and playing the cloud video stream, and improving the access concurrency of the dynamic and static fusion panoramic image.

[0138] The embodiment of the present application combines real-time monitoring video and digital master plan to generate a dynamic construction panorama and publish it to the H5 page, so that users can quickly locate and combine the monitoring screen with the real area. At the same time, the H5 web link can be conveniently and quickly accessed through a variety of devices, greatly reducing the threshold for using video surveillance.

[0139] In order to facilitate better implementation of the method for dynamic and static fusion of monitoring and construction digital master plan provided in the embodiment of the present application, the embodiment of the present application also provides a device based on the above-mentioned method for dynamic and static fusion of monitoring and construction digital master plan. The meaning of the terms is the same as that in the above-mentioned method for dynamic and static fusion of monitoring and construction digital master plan, and the specific implementation details can refer to the description in the embodiment of the method for dynamic and static fusion of monitoring and construction digital master plan.

[0140] The monitoring and construction digital master plan dynamic and static fusion method device in the embodiment of the present application has the function of implementing the monitoring and construction digital master plan dynamic and static fusion method provided in the above embodiment. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.

[0141] See also Figure 6 , Figure 6The structure diagram of the method and device for dynamic and static fusion of monitoring and construction digital master plan provided in the embodiment of the present application, the method and device for dynamic and static fusion of monitoring and construction digital master plan can be applied to a computing device in a scenario where content push is required. Specifically, the method and device 400 for dynamic and static fusion of monitoring and construction digital master plan may include a first acquisition module 401, a determination module 402, a second acquisition module 403, a transformation module 404, a mask module 405 and a fusion module 406, which are as follows:

[0142] The first acquisition module is used to acquire a digital construction master plan, which is a general layout of the construction site of the proposed project;

[0143] A determination module is used to determine the deployment points of the video surveillance of the construction site according to the digital construction master plan so that the video surveillance range covers the range of the digital construction master plan;

[0144] A second acquisition module: configured to acquire original video streams captured by the plurality of surveillance cameras after the plurality of surveillance cameras are deployed according to the deployment points;

[0145] Transformation module: used for performing perspective transformation operation on the original video stream to obtain a bird's-eye view video stream after perspective transformation;

[0146] Mask module: used for performing cropping and masking processing on the bird's-eye view video stream to obtain a bird's-eye view video stream to be spliced ​​that matches the construction digital master plan;

[0147] Fusion module: The bird's-eye view video stream to be stitched is stitched on the digital construction master plan to generate a dynamic and static fusion panoramic view of the construction site monitoring.

[0148] In one embodiment, the transformation module is specifically used to:

[0149] Transmitting the original video stream to a cloud server, wherein the cloud server includes a reading module and a perspective transformation module;

[0150] Reading the original video stream by the reading module to obtain original video source information;

[0151] Performing perspective transformation on the original video source information by the perspective transformation module to obtain perspective transformation information corresponding to the original video source information;

[0152] Sending the original video source information and the perspective transformation information to an edge server;

[0153] The edge server performs parsing operations on the original video source information and the perspective transformation information to obtain a bird's-eye view video stream after the perspective transformation of the original video stream.

[0154] In one embodiment, the original video source information includes multiple real-time frames, the perspective transformation information of the original video source information includes transformation matrix information when perspective transformation is performed on the multiple real-time frames, and the transformation module is further used to:

[0155] Taking one of the real-time frames as the target real-time frame, extracting the target feature points of the target real-time frame using the feature point extraction algorithm in the perspective view transformation module, and obtaining the original coordinates of the target feature points, wherein each of the target real-time frames corresponds to a target top-down view picture after perspective transformation;

[0156] Determine the target coordinates of the target feature point in the target top-down perspective image;

[0157] Calculate a target transformation matrix for perspective transformation of the target real-time frame according to the original coordinates and the target coordinates of the target feature points to obtain target transformation matrix information;

[0158] After obtaining the target transformation matrix information of the perspective transformation of each target real-time frame, the viewing angle perspective transformation information of the original video source information is obtained.

[0159] In one embodiment, the transformation module is further configured to:

[0160] The original coordinates and the target coordinates of the target feature points are substituted into a preset perspective transformation formula to solve the formula and obtain target transformation matrix information.

[0161] In one embodiment, the original video source information includes multiple real-time frames, the perspective transformation information of the original video source information includes transformation matrix information when perspective transformation is performed on the multiple real-time frames, and the transformation module is further used to:

[0162] Taking a real-time frame as a target real-time frame, the edge server calculates the top-view coordinates of all pixel coordinates of the target real-time frame in the target top-view image according to the target transformation matrix information corresponding to the target real-time frame, wherein each target real-time frame corresponds to a target top-view image after perspective transformation;

[0163] Generate a target overlooking perspective image according to the overlooking perspective coordinates of the target real-time frame and the target real-time frame;

[0164] After completing the generation of the target bird's-eye view picture of each target real-time frame, all the target bird's-eye view pictures are encoded to obtain a bird's-eye view video stream after the original video stream has been transformed by perspective.

[0165] In one embodiment, the mask module is specifically used for:

[0166] Simulating and splicing the multiple bird's-eye view video streams on the construction digital master plan;

[0167] Marking a repeated area formed by simulating and stitching the multiple top-view video streams as a mask area;

[0168] The masked area is cropped and masked using video mask technology to obtain multiple bird's-eye view video streams to be stitched together that match the construction digital master plan, wherein when the multiple bird's-eye view video streams to be stitched together are completely stitched together on the construction digital master plan, there are no repeated areas between any two bird's-eye view video streams to be stitched together.

[0169] In one embodiment, the monitoring and construction digital master plan dynamic and static fusion device is further used for:

[0170] Uploading the dynamic and static fusion panoramic image to a cloud server, and generating an H5 webpage link according to the dynamic and static fusion panoramic image;

[0171] If a request for browsing the H5 web page link is obtained from a terminal device, the dynamic and static fusion panoramic image is transmitted to the edge server in a P2P communication manner through the cloud server, and the edge server forwards the H5 web page link to the terminal device.

[0172] The embodiment of the present application integrates the surveillance video with the construction site master plan through perspective transformation, masking, pasting and other methods, constructs a dynamic panoramic view of the construction site from an intuitive overall perspective, achieves a dynamic satellite map-like digital supervision effect, and improves construction management efficiency.

[0173] The above describes the dynamic and static fusion device for monitoring and construction digital master plan in the embodiment of the present application from the perspective of modular functional entities. The following describes the dynamic and static fusion device for monitoring and construction digital master plan in the embodiment of the present application from the perspective of hardware processing.

[0174] It should be noted that Figure 7 The physical devices corresponding to the first acquisition module 401 and the second acquisition module 403 shown may be transceivers, radio frequency circuits, communication modules, and input / output (I / O) interfaces, etc., and the physical device corresponding to the transformation module 404 may be a processor.

[0175] Figure 7The devices shown can all have Figure 7 The structure shown, when Figure 7 The monitoring and construction digital master plan dynamic and static fusion method device shown has the following features: Figure 7 When the structure shown is Figure 7 The processor and the transceiver in the device can realize the same or similar functions as the transformation module 404 and the first acquisition module 401 provided in the device embodiment corresponding to the device, Figure 7 The memory storage processor in the computer program needs to be called when executing the above-mentioned monitoring and construction digital master plan dynamic and static fusion method.

[0176] When the computing device in the embodiment of the present application is a terminal device, the embodiment of the present application also provides a terminal device, such as Figure 7 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POs), a car computer, etc., taking the terminal device as a mobile phone as an example:

[0177] Figure 8 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Figure 8 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will understand that Figure 8 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0178] Combine the following Figure 8 A detailed introduction to the various components of the mobile phone:

[0179] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global system of mobile communication (Global system of mobile communication, GSM), General Packet Radio service (General Packet Radio service, GPRs), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, short messaging service (sMs), etc.

[0180] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0181] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0182] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Fig. 9 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0183] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0184] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0185] Wi-Fi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the Wi-Fi module 1070. It provides users with wireless broadband Internet access. Fig. 9 A Wi-Fi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0186] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, so as to monitor the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; optionally, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0187] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 1080 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.

[0188] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0189] In the embodiment of the present application, the processor 1080 included in the mobile phone also has the function of controlling and executing the above monitoring and construction digital master plan dynamic and static fusion method process executed by the monitoring and construction digital master plan dynamic and static fusion method device.

[0190] The present application also provides a server. Fig. 9 , Fig. 9 1 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (full name in English: central processing units, abbreviated in English: CPU) 1122 (for example, one or more processors) and memory 1132, one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1122 can be configured to communicate with the storage medium 1130 to execute a series of instruction operations in the storage medium 1130 on the server 1100.

[0191] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows server, Mac Os X, Unix, Linux, FreeBsD, etc.

[0192] The steps in the method for integrating dynamic and static monitoring and construction digital master plan in the above embodiment can be based on the Fig. 9 The structure of the server 1100 is shown.

[0193] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0194] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0195] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0196] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0197] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0198] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0199] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DsL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state hard disk solid state Disk (ssD)), etc.

[0200] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.

Claims

1. A method for integrating dynamic and static monitoring and construction digital master plan, characterized in that: The method for integrating dynamic and static monitoring and construction digital master plan includes: Obtaining a digital master plan of construction, wherein the digital master plan of construction is a general layout of the construction site of the proposed project; Determine the deployment points of the video surveillance of the construction site according to the digital construction master plan so that the video surveillance range covers the range of the digital construction master plan; After deploying the positions of the multiple surveillance cameras according to the deployment points, obtaining the original video streams captured by the multiple surveillance cameras; Performing a perspective transformation operation on the original video stream to obtain a bird's-eye view video stream after the perspective transformation; Performing cropping and masking processing on the bird's-eye view video stream to obtain a bird's-eye view video stream to be stitched that matches the construction digital master plan; The bird's-eye view video stream to be stitched is stitched on the digital construction master plan to generate a dynamic and static fusion panoramic view of the construction site monitoring.

2. The method for integrating dynamic and static monitoring and construction digital master plan according to claim 1 is characterized in that: The performing a perspective transformation operation on the original video stream to obtain a top-view video stream after perspective transformation, comprises: Transmitting the original video stream to a cloud server, wherein the cloud server includes a reading module and a perspective transformation module; Reading the original video stream by the reading module to obtain original video source information; Performing perspective transformation on the original video source information by the perspective transformation module to obtain perspective transformation information corresponding to the original video source information; Sending the original video source information and the perspective transformation information to an edge server; The edge server performs parsing operations on the original video source information and the perspective transformation information to obtain a bird's-eye view video stream after the perspective transformation of the original video stream.

3. The method for integrating dynamic and static monitoring and construction digital master plan according to claim 2 is characterized in that: The original video source information includes a plurality of real-time frames, the perspective transformation information of the original video source information includes transformation matrix information when perspective transformation is performed on the plurality of real-time frames, and the perspective transformation module performs perspective transformation on the original video source information to obtain the perspective transformation information corresponding to the original video source information, including: Taking one of the real-time frames as the target real-time frame, extracting the target feature points of the target real-time frame using the feature point extraction algorithm in the perspective view transformation module, and obtaining the original coordinates of the target feature points, wherein each of the target real-time frames corresponds to a target top-down view picture after perspective transformation; Determine the target coordinates of the target feature point in the target top-down perspective image; Calculate a target transformation matrix for perspective transformation of the target real-time frame according to the original coordinates and the target coordinates of the target feature points to obtain target transformation matrix information; After obtaining the target transformation matrix information of the perspective transformation of each target real-time frame, the viewing angle perspective transformation information of the original video source information is obtained.

4. The method for integrating dynamic and static monitoring and construction digital master plan according to claim 3 is characterized in that: The step of calculating a target transformation matrix for perspective transformation of the target real-time frame according to the original coordinates and the target coordinates of the target feature points to obtain target transformation matrix information includes: The original coordinates and the target coordinates of the target feature points are substituted into a preset perspective transformation formula to solve the formula and obtain target transformation matrix information.

5. The method for integrating dynamic and static monitoring and construction digital master plan according to claim 2 is characterized in that: The original video source information includes a plurality of real-time frames, the perspective transformation information of the original video source information includes transformation matrix information when perspective transformation is performed on the plurality of real-time frames, and the edge server performs a parsing operation on the original video source information and the perspective transformation information to obtain a bird's-eye view video stream after the perspective transformation of the original video stream, including: Taking a real-time frame as a target real-time frame, the edge server calculates the top-view coordinates of all pixel coordinates of the target real-time frame in the target top-view image according to the target transformation matrix information corresponding to the target real-time frame, wherein each target real-time frame corresponds to a target top-view image after perspective transformation; Generate a target overlooking perspective image according to the overlooking perspective coordinates of the target real-time frame and the target real-time frame; After completing the generation of the target bird's-eye view picture of each target real-time frame, all the target bird's-eye view pictures are encoded to obtain a bird's-eye view video stream after the original video stream has been transformed by perspective.

6. The method for integrating dynamic and static monitoring and construction digital master plan according to claim 1 is characterized in that: The step of performing cropping and masking processing on the bird's-eye view video stream to obtain a bird's-eye view video stream to be stitched that matches the construction digital master plan includes: Simulating and splicing the multiple bird's-eye view video streams on the construction digital master plan; Marking a repeated area formed by simulating and stitching the multiple top-view video streams as a mask area; The masked area is cropped and masked using video mask technology to obtain multiple bird's-eye view video streams to be stitched together that match the construction digital master plan, wherein when the multiple bird's-eye view video streams to be stitched together are completely stitched together on the construction digital master plan, there are no repeated areas between any two bird's-eye view video streams to be stitched together.

7. The method for integrating dynamic and static monitoring and construction digital master plan according to claim 1 is characterized in that: After the bird's-eye view video streams to be stitched are stitched together on the digital construction master plan to generate a dynamic and static fusion panoramic view of the construction site monitoring, the method further includes: Uploading the dynamic and static fusion panoramic image to a cloud server, and generating an H5 webpage link according to the dynamic and static fusion panoramic image; If a request for browsing an H5 web page link is obtained from a terminal device, the dynamic and static fusion panoramic image is transmitted to the edge server in a P2P communication manner through the cloud server, and the edge server forwards the H5 web page link to the terminal device.

8. A dynamic and static fusion device for monitoring and construction digital master plan, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for dynamic and static fusion of monitoring and construction digital master plan as described in any one of claims 1 to 7 when executing the computer program.

9. A computing device, characterized in that It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for dynamic and static fusion of monitoring and construction digital master plan as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: It includes instructions, which, when executed on a computer, enable the computer to execute the method for dynamic and static fusion of monitoring and construction digital master plan as described in any one of claims 1 to 7.

Images