Production intelligent visual holographic AR monitoring system and monitoring method

Abstract

The invention relates to the technical field of monitoring systems, and discloses a production intelligent visual holographic AR monitoring system and method, and the system comprises a digital map construction module, a monitoring camera calibration module, a monitoring video stream calling module, a monitoring video image and map scene fusion module, and a data real-time display module. The real-time intelligent analysis and data visualization module is used for real-time intelligent analysis and data visualization of a power grid measurement monitoring scene; wherein the digital map construction module is used for collecting and processing related geographic space data, and the monitoring camera calibration module is used for calibrating each monitoring camera; according to the system, intelligent visual processing, a holographic technology and an AR technology are combined, the monitoring efficiency and the intelligent level of a production site are greatly improved, and the production safety and the management efficiency of a metering center are ensured. Meanwhile, the system provides visual and convenient operation experience for operators through real-time data display and voice interaction, and further promotes the intelligent process of electric power measurement management.

Description

Technical Field

[0001] The present invention relates to the technical field of monitoring systems, and in particular to the production of intelligent visual holographic AR monitoring systems and monitoring methods. Background Art

[0002] Intelligent Vision is a new visual monitoring technology that integrates computer vision, digital twins, surveillance video streams, camera calibration, SLAM technology (simultaneous localization and mapping), and other high-tech technologies. It takes real vision as the core and deeply integrates and presents multi-dimensional information. With almost no human intervention, it can locate, identify and track targets in digital video images taken by surveillance cameras, intelligently present relevant business data information on the monitoring screen, and analyze and judge the target's behavior and status on this basis. It can not only complete daily production management and monitoring work, but also proactively discover abnormal problems and respond in a timely manner.

[0003] Intelligent visual technology is based on the latest cutting-edge visualization concept of "real site + data presentation". Combined with the gain of the IoT perception layer and network layer, it can present the on-site conditions 100% realistically, and realize "highly transparent" monitoring and management of the site. It can reflect the production status in the most real, direct, timely and accurate way, and quickly discover and solve problems.

[0004] By using the video surveillance images of the single-phase assembly line of the metrology center and combining them with real-time production data, we can create intelligent visual monitoring technology for the single-phase assembly line, which can realize intelligent visual monitoring of the working status, calibration progress, calibration results of the assembly line of the metrology center, and the types and status statistics of inventory materials in the intelligent stereoscopic warehouse, thereby combining the background data with the real scene. It can not only highlight the interactive advantages of three-dimensional data, but also make up for the shortcomings of the limited field of vision of real-scene monitoring. It realizes "seeing the data dimension" in the real physical space and "seeing the full-dimensional information" for the real scene.

[0005] This application proposes a production intelligent visual holographic AR monitoring system and monitoring method, aiming to combine traditional monitoring technology with advanced intelligent visual processing, holographic technology and AR augmented reality technology to provide a highly integrated and intelligent monitoring solution for the power grid company's metering production management platform. Summary of the invention

[0006] The purpose of the present invention is to provide a production intelligent visual holographic AR monitoring system and a monitoring method to solve the problems raised in the above background technology.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] Produce intelligent visual holographic AR monitoring system and monitoring method, including digital map construction module, monitoring camera calibration module, monitoring video stream call module, monitoring video screen and map scene fusion module, and real-time data display module, which are used for real-time intelligent analysis and data visualization of power grid metering monitoring scenes;

[0009] Among them, the digital map construction module is used to collect and process relevant geographic spatial data, the surveillance camera calibration module is used to calibrate each surveillance camera, the surveillance video stream calling module is used to obtain the real-time video stream from the surveillance camera calibration module and transmit it to the data processing module for processing, the surveillance video screen and map scene fusion module is responsible for fusing the processed video stream with the generated map data to generate a complete surveillance scene, and the real-time data display module is responsible for displaying the generated surveillance scene in real time so that the operator can intuitively understand the monitoring situation.

[0010] Preferably, the system also includes a voice interaction module, allowing the user to switch monitoring screens and perform data query operations through voice commands.

[0011] Preferably, the monitoring video screen and map scene fusion module uses a distortion algorithm shader to simulate physical camera distortion to achieve accurate fusion of the monitoring screen and the digital map.

[0012] Preferably, the real-time data display module displays the real-time status and verification progress information of the functional unit through AR glasses.

[0013] Preferably, the monitoring method for producing an intelligent visual holographic AR monitoring system comprises the following steps:

[0014] a) Use the digital map construction module to establish a digital map of the single-phase electric energy meter calibration production line, collect layout information and equipment dimensions through on-site sampling, and use 3D modeling software to construct points, lines, and surfaces to form a digital 3D model;

[0015] b) implementing single-view vanishing point calibration of the surveillance camera through the surveillance camera calibration module;

[0016] c) The monitoring video stream calling module calls the monitoring video stream, accesses the intelligent visual monitoring system through the video aggregation platform or NVR server, and displays the monitoring area in real time;

[0017] d) The monitoring video screen and map scene fusion module adds distortion algorithm processing to ensure that the monitoring video screen and the digital map scene are correctly integrated, overcoming the influence of camera distortion, and displaying the status and verification progress information of the single-phase pipeline functional unit in real time in the AR glasses, realizing the intuitive combination of data and real objects;

[0018] f) The real-time data display module displays the specific data of the single-phase electricity meter calibration process, including calibration items, process values, and conclusions, to improve the precision and efficiency of management.

[0019] Preferably, the surveillance camera calibration module uses a single-view vanishing point technology for precise calibration to ensure that the virtual camera's parameters such as field of view angle and focal length match the actual physical camera parameters, thereby achieving synchronous observation of virtual and real scenes.

[0020] Preferably, the calling and integration of the monitoring video stream calling module transmits the monitoring screen to the intelligent visual monitoring program in real time through a specific platform or server, thereby realizing remote monitoring and access to real-time monitoring screens.

[0021] Preferably, the digital map construction model also includes high-precision simulation of ambient lighting and material texture, ensuring a high degree of similarity between the digital map and the actual environment, providing a more realistic visual experience for AR monitoring.

[0022] Preferably, the monitoring video screen and map scene fusion module also includes a dynamic perspective matching mechanism, which automatically adjusts the perspective and position of the virtual camera according to the real-time movement or adjustment of the physical camera, so as to achieve seamless connection between the virtual scene and the physical world at any perspective.

[0023] Preferably, the method also includes a voice interactive intelligent application function, allowing users to operate the system and query information through voice commands, thereby enhancing the human-computer interaction experience.

[0024] The present invention has at least the following beneficial effects:

[0025] This solution uses the monitoring video stream calling module to obtain real-time video streams from the monitoring camera calibration module and transmit it to the data processing module for processing. The monitoring video screen and map scene fusion module is responsible for fusing the processed video stream with the generated map data to generate a complete monitoring scene, realizing the combination of intelligent visual processing, holographic technology and AR augmented reality technology, greatly improving the monitoring efficiency and intelligence level of the production site, and ensuring the production safety and management efficiency of the metering center. At the same time, the system provides operators with an intuitive and convenient operating experience through real-time data display and voice interaction, further promoting the intelligent process of power metering management. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 It is a system structure block diagram of the present invention;

[0028] Figure 2 It is a schematic diagram of the intelligent visual monitoring effect of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1-Figure 2 The present invention provides a technical solution: to produce an intelligent visual holographic AR monitoring system and a monitoring method, including a digital map construction module, a monitoring camera calibration module, a monitoring video stream call module, a monitoring video screen and map scene fusion module, and a real-time data display module, which are used for real-time intelligent analysis and data visualization of power grid metering monitoring scenes;

[0031] Among them, the digital map construction module is used to collect and process relevant geospatial data, the monitoring camera calibration module is used to calibrate each monitoring camera to ensure that the images it captures are clear, stable and accurate, the monitoring video stream calling module is used to obtain the real-time video stream from the monitoring camera calibration module and transmit it to the data processing module for processing, the monitoring video screen and map scene fusion module is responsible for fusing the processed video stream with the generated map data to generate a complete monitoring scene, and the data real-time display module is responsible for displaying the generated monitoring scene in real time so that the operator can intuitively understand the monitoring situation; the monitoring video screen and map scene fusion module uses a distortion algorithm shader to simulate the distortion of the physical camera to achieve accurate fusion of the monitoring screen and the digital map; the data real-time display module displays the real-time status and calibration progress information of the functional unit through AR glasses

[0032] In some embodiments, the system also includes a voice interaction module, allowing the user to switch monitoring screens and perform data query operations through voice commands.

[0033] In some embodiments, a monitoring method for producing an intelligent visual holographic AR monitoring system includes the following steps:

[0034] a) Use the digital map construction module to establish a digital map of the single-phase electric energy meter calibration production line, collect layout information and equipment dimensions through on-site sampling, use 3D modeling software to construct points, lines and surfaces, and form a digital 3D model to provide basic data for intelligent monitoring;

[0035] Through on-site sampling, the layout information of the single-phase electric energy meter calibration line, the size information of each functional unit component, the size information of turnover boxes, pallets, electric energy meters and other equipment are collected, and a digital map is created. First, on-site sampling is carried out on the single-phase electric energy meter calibration line. The layout information of the on-site assembly line and the size information of each functional unit component are collected, and the size information of turnover boxes, pallets, electric energy meters and other equipment is collected to provide data support for the creation of digital maps.

[0036] Import the CAD model file into the 3D modeling software, and use the Grid and Snap function, mesh modeling, and polygon modeling technology of the 3D modeling software to construct points, lines, and surfaces of the 3D digital model to create a digital 3D model of each functional unit of the assembly line.

[0037] By digitally modeling the single-phase assembly line area, a scene map of the area is constructed, and the scene map is imported into the intelligent visual monitoring program. As the background data of intelligent visual monitoring, the scene map is not directly displayed in the display area of ​​intelligent vision, but it is the most important basis for intelligent vision to identify functional units.

[0038] b) Implementing single-view vanishing point calibration of surveillance cameras through the surveillance camera calibration module, using the Robert Collins algorithm to calculate physical parameters to ensure consistency between the virtual camera view and the actual surveillance image;

[0039] After building a 3D digital model of the electric energy meter calibration line, we need to calibrate each camera in this model. The calibration process uses the single view vanishing point calibration technology, such as the Robert T Collins algorithm. These calibration parameters will be used to calculate the position, viewing angle, focal length and other information of the camera in the actual scene. These parameters will be used for subsequent video processing and matching.

[0040] The single-view vanishing point calibration technology, Robert T Collins algorithm, is used to calibrate the surveillance cameras of the single-phase electricity meter calibration line of the Provincial Metering Center in the digital twin scenario, and the calibration parameters such as FOV, focal length, CMOS size, spatial position, rotation angle, center offset, longitudinal and lateral tilt angles of the physical surveillance cameras are calculated.

[0041] Use these calibration parameters to create a virtual camera in the intelligent visual monitoring program, so that the content viewed by the virtual camera in the intelligent visual map scene is consistent with the content viewed by the physical monitoring camera in the physical space. By establishing a virtual camera for the single-phase electric energy meter calibration line, the one-to-one mapping of each functional unit in the on-site monitoring video stream of the area and each functional unit in the digital twin scene is achieved, and finally the deep integration of the intelligent visual map scene and the on-site monitoring screen is achieved, providing a foundation for the realization of intelligent visual monitoring.

[0042] c) The monitoring video stream calling module calls the monitoring video stream, accesses the intelligent visual monitoring system through the video aggregation platform or NVR server, and displays the monitoring area in real time;

[0043] Through the video aggregation platform or network video recording device (NVR server), the camera video stream of the electric energy meter calibration production line area is connected to our intelligent visual monitoring system. In this way, the content captured by the camera can be viewed on the monitoring screen, and the content can be analyzed through image recognition technology;

[0044] That is: through the video aggregation platform or NVR server, the video stream of the surveillance camera in the single-phase electric energy meter calibration production line area is connected to the intelligent visual monitoring system, and the on-site monitoring screen access of key areas is completed, providing the front-end display screen for intelligent visual monitoring.

[0045] d) The monitoring video screen and map scene fusion module adds distortion algorithm processing to ensure that the monitoring video screen and the digital map scene are correctly integrated, overcoming the influence of camera distortion, and displaying the status and verification progress information of the single-phase pipeline functional unit in real time in the AR glasses, realizing the intuitive combination of data and real objects;

[0046] Usually, there will be serious camera distortion in the surveillance video. Camera distortion refers to the phenomenon that the image is deformed or distorted during imaging due to the characteristics of the camera's optical system. In order to ensure the perfect integration of the surveillance video and the digital map scene, it is necessary to add distortion effects to the virtual camera in the digital map. There may be some camera distortion problems in the video. In order to avoid this problem affecting our understanding of the scene, we need to add corresponding distortion algorithm shaders to each camera in the digital map of the power meter calibration pipeline. In this way, we can ensure that no matter which camera is viewed from, we can get a real and accurate image.

[0047] Add a distortion algorithm shader to the virtual camera, add the focal length, principal point position, radial distortion, tangential distortion and other parameters obtained during the physical camera calibration to the distortion algorithm, and use the distortion algorithm to simulate the distortion of the digital map image rendered by the virtual camera to achieve the fusion display of the digital map scene and the surveillance video image.

[0048] When the AR glasses recognize the functional units, the system automatically extracts relevant data from the intelligent operation and maintenance platform, and displays it on the physical functional units through AR technology, providing real-time production management information: the single-phase assembly line calibration process data display, the system can retrieve the detailed calibration records of the electric energy meter from the database, and display them intuitively through AR glasses, which is convenient for on-site monitoring and decision-making.

[0049] f) The real-time data display module displays the specific data of the single-phase electricity meter calibration process, including calibration items, process values, and conclusions, to improve the precision and efficiency of management.

[0050] When the intelligent visual monitoring system detects a functional unit in a single-phase assembly line, we will also receive relevant data feedback. This data may come from the message bus of the intelligent operation and maintenance platform, or from the data table of the electric energy meter verification process. This data is presented on the screen of AR glasses in the form of data visualization, so that users can better understand and manage the working status of the single-phase assembly line.

[0051] When the AR glasses recognize the functional units in the single-phase assembly line area, the AR monitoring program obtains the status, calibration plan, calibration progress, calibration parameters and other data of the corresponding functional units from the intelligent operation and maintenance platform message bus, and uses data visualization technology to superimpose and integrate the functional units' real-time status, current calibration progress, current calibration plan and other calibration information on the physical functional units in the AR glasses.

[0052] When the AR glasses recognize the functional units in the single-phase assembly line area, the AR monitoring program obtains the data such as the inspected items, the inspection process of the inspected items, and the inspection conclusions of the selected electric energy meter from the electric energy meter inspection process data table of the intelligent operation and maintenance platform, and displays the meter barcode, inspected item list, inspected item inspection parameters, inspected item inspection process values, inspected item inspection conclusions and other information of the electric energy meter on the physical functional unit in the AR glasses by superimposing and integrating them with data visualization technology.

[0053] In some embodiments, the surveillance camera calibration module uses single-view vanishing point technology for precise calibration to ensure that the field of view angle, focal length and other parameters of the virtual camera match the actual physical camera parameters, thereby achieving synchronous observation of virtual and real scenes.

[0054] Furthermore, the calling and integration of the monitoring video stream calling module transmits the monitoring screen to the intelligent visual monitoring program in real time through a specific platform or server, thereby realizing the access of remote monitoring and real-time monitoring screen.

[0055] Furthermore, the digital map construction model also includes high-precision simulation of environmental lighting and material texture, ensuring a high degree of similarity between the digital map and the actual environment, providing a more realistic visual experience for AR monitoring.

[0056] In some embodiments, the surveillance video image and map scene fusion module also includes a dynamic perspective matching mechanism, which automatically adjusts the perspective and position of the virtual camera according to the real-time movement or adjustment of the physical camera, so as to achieve seamless connection between the virtual scene and the physical world at any perspective.

[0057] The method also includes a voice interactive intelligent application function, which allows users to operate the system and query information through voice commands, enhance the human-computer interaction experience, and improve work efficiency and safety. It includes a processor and a memory. The processor is used to receive voice commands from the user and perform corresponding operations; the memory is used to store information such as the status information and processing results of the smart device.

[0058] When the user issues a voice command, the smart device will recognize the voice command input by the user, and update and modify the status information and processing results of the smart device accordingly based on the recognition results. For example, if the user wants to turn on the camera function of the smart device, just say "take a photo", and the smart device will start the camera application and perform corresponding operations based on the recognition results. If the user wants to close an application on the smart device, just say "exit", and the smart device will stop the application and return to the previous interface based on the recognition results. In addition to basic voice commands, smart devices also support the execution of compound commands. For example, if the user wants to check the battery status of the smart device, he can issue a compound command of "check battery", and the smart device will provide the user with information about the battery status after the recognition results.

[0059] In addition, smart devices also provide multi-language support, allowing users from different countries and regions to use smart devices conveniently.

[0060] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0061] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Produce intelligent visual holographic AR monitoring system, characterized by: It includes a digital map construction module, a surveillance camera calibration module, a surveillance video stream call module, a surveillance video screen and map scene fusion module, and a real-time data display module, which are used for real-time intelligent analysis and data visualization of power grid metering monitoring scenarios; Among them, the digital map construction module is used to collect and process relevant geographic spatial data, the surveillance camera calibration module is used to calibrate each surveillance camera, the surveillance video stream calling module is used to obtain the real-time video stream from the surveillance camera calibration module and transmit it to the data processing module for processing, the surveillance video screen and map scene fusion module is responsible for fusing the processed video stream with the generated map data to generate a complete surveillance scene, and the real-time data display module is responsible for displaying the generated surveillance scene in real time so that the operator can intuitively understand the monitoring situation.

2. The production intelligent visual holographic AR monitoring system according to claim 1 is characterized by: The system also includes a voice interaction module that allows users to switch monitoring screens and query data through voice commands.

3. The production intelligent visual holographic AR monitoring system according to claim 1 is characterized by: The monitoring video screen and map scene fusion module adopts a distortion algorithm shader to simulate physical camera distortion, thereby achieving accurate fusion of the monitoring screen and the digital map.

4. The production intelligent visual holographic AR monitoring system according to claim 1 is characterized by: The real-time data display module displays the real-time status and verification progress information of the functional unit through AR glasses.

5. The monitoring method for producing an intelligent visual holographic AR monitoring system according to any one of claims 1 to 4, characterized in that: The following steps are involved: a) Use the digital map construction module to establish a digital map of the single-phase electric energy meter calibration production line, collect layout information and equipment dimensions through on-site sampling, and use 3D modeling software to construct points, lines, and surfaces to form a digital 3D model; b) implementing single-view vanishing point calibration of the surveillance camera through the surveillance camera calibration module; c) The monitoring video stream calling module calls the monitoring video stream, accesses the intelligent visual monitoring system through the video aggregation platform or NVR server, and displays the monitoring area in real time; d) The monitoring video screen and map scene fusion module adds distortion algorithm processing to ensure that the monitoring video screen and the digital map scene are correctly integrated, overcoming the influence of camera distortion, and displaying the status and verification progress information of the single-phase pipeline functional unit in real time in the AR glasses, realizing the intuitive combination of data and real objects; e) The real-time data display module displays the specific data of the single-phase electricity meter calibration process, including calibration items, process values, and conclusions, to improve the precision and efficiency of management.

6. The monitoring method for producing an intelligent visual holographic AR monitoring system according to claim 5, characterized in that: The surveillance camera calibration module uses the single-view vanishing point technology for precise calibration, ensuring that the field of view angle, focal length and other parameters of the virtual camera match the actual physical camera parameters, thereby achieving synchronous observation of virtual and real scenes.

7. The monitoring method for producing an intelligent visual holographic AR monitoring system according to claim 5, characterized in that: The calling and integration of the monitoring video stream calling module transmits the monitoring screen to the intelligent visual monitoring program in real time through a specific platform or server, thereby realizing the access of remote monitoring and real-time monitoring screen.

8. The monitoring method for producing an intelligent visual holographic AR monitoring system according to claim 5, characterized in that: The digital map construction model also includes high-precision simulation of environmental lighting and material texture, ensuring a high degree of similarity between the digital map and the actual environment, providing a more realistic visual experience for AR monitoring.

9. The monitoring method for producing an intelligent visual holographic AR monitoring system according to claim 5, characterized in that: The monitoring video screen and map scene fusion module also includes a dynamic perspective matching mechanism, which automatically adjusts the perspective and position of the virtual camera according to the real-time movement or adjustment of the physical camera, so as to achieve seamless connection between the virtual scene and the physical world at any perspective.

10. The monitoring method for producing an intelligent visual holographic AR monitoring system according to claim 5, characterized in that: The method also includes a voice interactive intelligent application function, which allows users to operate the system and query information through voice commands, thereby enhancing the human-computer interaction experience.

Images