A device control system and method from geographic space to high-precision digital space

Abstract

The present invention provides an equipment control system and method from geographic space to high-precision digital space, including: constructing industrial production scenes and electromechanical equipment models with the same structural features as the production site and control methods to form a digital twin system; constructing a video stream system to continuously output production scene video stream data based on orthographic projection, geometric correction, and panoramic stitching images and audio fusion in real time. Based on the digital twin and video stream system, a multimedia control platform integrating graphics, images, videos, audio, monitoring and production data is constructed to achieve a control mode of mutual supplementation and verification of production information; remote immersive digital inspection and disaster warning of industrial production sites are realized; remote control and fault diagnosis of electromechanical equipment on industrial sites are realized. The platform will improve the expression range and accuracy of digital twins, and realize remote precision control and immersive inspection of smart mines, smart factories and other places, as well as production goals with few or no people.

Description

Technical Field

[0001] The present invention relates to the field of video stitching, digital twins and space-time scene establishment of industrial production sites in smart mines and smart factories, and in particular to a system and method for equipment management and control from geographic space to high-precision digital space. Background Art

[0002] At present, with the continuous advancement of video splicing and digital twin technology, people's demand for the perception of the conditions of scenes and electromechanical equipment in industrial production sites is constantly expanding, especially through information, automation, and intelligent methods and systems to automatically identify the hazardous source information of objects such as personnel, environment, and equipment, and realize the safety management and control of important industrial sites and high-risk operation scenes. For example, coal mining working faces, natural gas transmission pipelines, nuclear power plant reactors, chemical plant storage tank areas and other high-risk places.

[0003] In response to the above-mentioned high-risk scene operation problems, most of them use video surveillance combined with various sensor indicators to monitor the scene. This type of method can give early warnings to some dangerous signals in the scene to a certain extent, but there are problems such as local visibility and low acquisition density and accuracy. It cannot effectively solve the problems of lack of spatiotemporal attributes of video information and lack of digital twin fusion information in the monitoring and early warning system. The main problems are as follows:

[0004] (1) The video has a large deformation, and no geometric correction or image stitching has been performed. The video pixels have no spatial geographic coordinate information, and the temporal and spatial information of the objects in the scene cannot be truly and accurately restored;

[0005] (2) The object expression accuracy of the digital twin system is related to the modeling accuracy and the amount of collected data or perceived information. The expression accuracy often fails to meet actual requirements, and local spatial object events cannot be reflected, such as roof leakage and rib spalling in the fully mechanized mining face of a coal mine, and local damage to factory equipment.

[0006] (3) Videos can truly reflect the surface features of industrial scenes and electromechanical equipment, while digital twins can truly reflect the internal features of industrial scenes and electromechanical equipment. The fusion of videos and digital twins that do not have spatiotemporal data attributes is more difficult.

[0007] (4) Mainstream industrial control technologies lack immersive inspection and visual remote control capabilities. Summary of the invention

[0008] In view of the above problems, the present invention proposes a device management and control system and method from geographic space to high-precision digital space.

[0009] The embodiment of the present invention provides a device management and control system from geographic space to high-precision digital space, including: a digital twin system, a video streaming system, a multimedia digital space system, and an industrial control system;

[0010] The digital twin system includes: an industrial production scene model, an electromechanical equipment model, a script description, and an information feedback function module, which is used to provide digital twin data of industrial production scenes and electromechanical equipment;

[0011] The video stream system includes: a panoramic camera device, image stitching, orthographic projection, geometric correction, and a pixel three-dimensional geographic coordinate assignment function module, which is used to construct a vector space and output production scene video stream data based on orthographic projection, geometric correction, and panoramic stitching images and audio fusion in real time and continuously;

[0012] The multimedia digital space system is used to fuse the digital twin data and the video stream data to form high-precision fused data that integrates real-time graphics, images, videos, and audios, and use an information correction module to perform spatiotemporal synchronization correction on the fused data to output multimedia digital space data;

[0013] The industrial control system is used to combine the multimedia digital space system based on the multimedia digital space data, and to construct a multimedia management and control platform using industrial control technology or system;

[0014] The multimedia management and control platform is used to realize remote control and fault diagnosis of electromechanical equipment at industrial production sites, realize digital inspections and disaster warnings of industrial production scenes, and realize the mutual supplementation and verification of production information by the digital twin system and the video streaming system. Management and control mode.

[0015] Optionally, the structural features of the industrial production scene model and the electromechanical equipment model are the same as the structural features of the scenes and equipment at the industrial site;

[0016] The structural features include: internal and external structure, material, spatial position, geometric constraints, appearance mapping, and degrees of freedom;

[0017] The information feedback function module is used to collect industrial scene and electromechanical equipment structure and operation characteristic information;

[0018] The script description module adopts a visual script programming method and utilizes feature information to drive industrial scenes and electromechanical equipment to update, move, and diagnose faults, so as to realize virtual-real, virtual-real, and virtual-real symbiosis.

[0019] Optionally, the video streaming system outputs real-time orthographic projection, geometric correction, panoramic stitching image and audio fusion video streaming data of the industrial production scene and the electromechanical equipment in real time and continuously;

[0020] The video stream data is a real-time, continuous data sequence, which includes: real-time image data, audio data, three-dimensional geographic coordinates assigned to pixels, and camera control instruction information;

[0021] The video streaming system is configured to have the functions of audio and video feature information extraction, model training, analysis, and recognition result prompting.

[0022] Optionally, the digital twin system is used to collect known information of the physical environment of industrial production in real time, so as to express and control the known information in real time;

[0023] The video stream system is used to supplement the information that the digital twin system cannot collect;

[0024] The two types of spatiotemporal object information of the digital twin system and the video stream system are combined to complement, verify and check the expression accuracy of the information.

[0025] Optionally, the multimedia digital space system outputs multimedia digital space data, combining and displaying the scenes of the industrial production site and the structural characteristics of electromechanical equipment, operation and fault characteristics, personnel behavior and environmental disaster identification results and linkage disposal results;

[0026] Among them, the combined display is a data display and expression method of human-computer interaction configuration, so that the digital twin system and the video stream system can display the scenes and information of electromechanical equipment at the industrial production site separately or synchronously. The synchronization means: according to the direction, position, translation and zoom parameters of the observer's line of sight, the same production scene and related information are displayed at the same time.

[0027] Optionally, the method for the multimedia digital space system to fuse the digital twin data and the video stream data includes:

[0028] Step S1: The multimedia digital space system marks the unique numbers (n) of all camera points in the video stream system. 1 ,n 2 ,n 3 ...) and coordinates ((x v1 ,y v1 , z v1 ), (x v2 ,y v2 , z v2 ), (x v3 ,y v3 , z v3 )……), storing information such as camera point numbers and coordinates in the database of the digital twin system;

[0029] Step S2: The multimedia digital space system calibrates the camera point position in the digital twin system and obtains the same number (n 1 ,n 2 ,n 3 ……) The coordinates of the camera point ((x s1 ,y s1 ,z s1 ),(x s2 ,y s2 ,z s2 ),(x s3 ,y s3 ,z s3 )……);

[0030] Step S3: Based on the two sets of coordinates, a mathematical model for mutual geometric correction is established to unify the two coordinate systems; if the unified coordinate system is based on the digital twin system, the three-dimensional coordinates of all pixels of the video stream system are corrected to the digital twin coordinate system. If the unified coordinate system is based on the video stream system, the three-dimensional coordinates of all pixels of the digital twin system are corrected to the video stream coordinate system;

[0031] Step S4: The digital twin system acquires the structure, operation and fault, personnel behavior and environmental data of the scene and electromechanical equipment model of the industrial production site in real time and continuously from the industrial production site. The multimedia digital space system uses the information correction module to associate the structure, operation and fault, personnel behavior and environmental data of the scene and electromechanical equipment model with the camera point number and coordinates (x, y, z), panoramic orthographic projection stitching image and audio according to the time and space relationship to form fused data;

[0032] Step S5: the digital twin system queries the numbers and coordinates of all camera points, arranges the numbers in the order of inspection time or random positions, combines them into an inspection route in the industrial production scene, and stores them in the database of the digital twin system;

[0033] Step S6: The digital twin system queries the information of the camera point closest to the observer's current position in the spatial distance according to the set route or the human-computer interaction random inspection process, and constrains the digital twin scene viewpoint to the camera point, generates a combination of a 360°×360° digital twin scene and a panoramic orthographic projection stitching image, and fuses the display device monitoring information to the digital twin and video stream image of the corresponding position;

[0034] Step S7: During the inspection process along the set route, the digital twin system queries the observer's current position, sight direction, translation amount, and zoom amount, and publishes them to the video streaming system. The video streaming system subscribes to the information in real time and adjusts the observer's visual parameters to maintain the same position, sight direction, translation amount, and zoom amount as the digital twin system.

[0035] Step S8: The multimedia digital space system accurately expresses the scene of the industrial production site and the audio and video features of the electromechanical equipment model and the operation information of the electromechanical equipment in real time and traceably by changing the transparency, brightness and contrast of the panoramic orthographic projection stitching image of the video stream system and fusing the fused data within the observation window.

[0036] Step S9: The multimedia digital space system accurately expresses the scenes and electromechanical equipment characteristics of the industrial production site in real time and traceably by hiding and displaying the scenes and electromechanical equipment models of the industrial production site in the digital twin system.

[0037] Optionally, the method for combining the industrial control system with the multimedia digital space system includes:

[0038] Step T1: Based on the multimedia digital space system, query and display the structure, operation and failure, personnel behavior and environmental data of the industrial production site scene and electromechanical equipment model;

[0039] Step T2: query the disaster warning, remote control and fault category parameters of the industrial production site, and perform disaster warning and fault diagnosis on the industrial production site;

[0040] Step T3: Based on the scenes of the industrial production site and the characteristic data of the electromechanical equipment of the multimedia digital space system, an industrial control logic is established to control the electromechanical equipment to move accurately according to the predetermined actions and positions;

[0041] Step T4: Displaying real-time images and audio of the industrial production site in the three-dimensional scene of the multimedia digital space system, and timely triggering the shutdown and locking of the industrial control system when danger occurs through human eye observation or video AI diagnosis;

[0042] Step T5: After the industrial control system controls the action and locking linkage of the electromechanical equipment on the industrial site, the information is fed back to the digital twin system, and the system interface synchronously displays the equipment operation status characteristics.

[0043] Optionally, after the industrial control system executes the control actions and locking linkages of the electromechanical equipment at the industrial production site, it feeds back the information to the digital twin system to enable operators to remotely immerse in digital space inspections and remote control of dangerous areas and electromechanical equipment at the industrial site.

[0044] Optionally, the fusion of the respective data of the digital twin system and the video stream system is a mutual complement and verification of production information to achieve interoperability, real-time, scalability, fidelity and closed-loop;

[0045] The interoperability refers to: the two-way, dynamic interaction and real-time connection between the digital twin system and the video streaming system and the physical objects of the industrial production scene and electromechanical equipment, and the continuous mutual complementation, verification and control of the spatiotemporal data of the digital twin system and the video streaming system;

[0046] The real-time performance means that the digital twin data and the video stream data are spatiotemporally synchronized based on a unified clock and spatial coordinate system;

[0047] The scalability means that the industrial production scenario model and electromechanical equipment model have the ability to be integrated, added and replaced, and the model content can be expanded;

[0048] The fidelity means that the industrial production scene model and electromechanical equipment model maintain high-fidelity simulation of the geometric structure of the physical entity while also simulating real-time graphics, images, videos, audio, states, phases and tenses to ensure high similarity with the physical entity;

[0049] The closed-loop nature means that the multimedia management and control platform is used to describe the visual model and internal mechanism of the physical entity at the industrial production site, and utilizes the data of the physical entity to conduct analysis and reasoning, optimize process parameters and operating parameters, and realize decision-making functions, that is, to use a closed-loop system for the digital model and the physical entity.

[0050] In a second aspect, an embodiment of the present invention provides a device control method from a geographic space to a high-precision digital space, the device control method comprising:

[0051] Constructing a digital twin system, the digital twin system comprising: an industrial production scene model, an electromechanical equipment model, a script description, and an information feedback function module, for providing digital twin data of industrial production scenes and electromechanical equipment;

[0052] Constructing a video streaming system, the video streaming system comprising: a panoramic camera device, an image stitching, an orthographic projection, a geometric correction, and a pixel three-dimensional geographic coordinate assignment function module, for constructing a vector space and real-time and continuous output of production scene video streaming data based on orthographic projection, geometric correction, and panoramic stitching images and audio fusion;

[0053] Construct a multimedia digital space system, which is used to fuse the digital twin data and the video stream data to form high-precision fused data that integrates real-time graphics, images, videos, and audios, and use an information correction module to perform spatiotemporal synchronization correction on the fused data to output multimedia digital space data. The multimedia digital space system is configured to combine and display the digital twin data and video stream data as needed, and displays the scenes of the industrial production site and the information of electromechanical equipment separately or synchronously;

[0054] Constructing an industrial control system, the industrial control system is used to combine the multimedia digital space system based on the multimedia digital space data, and to construct a multimedia management and control platform using industrial control technology or system; the multimedia management and control platform is used to realize remote control and fault diagnosis of electromechanical equipment at industrial production sites, realize digital inspection and disaster warning of industrial production scenes, and realize the mutual supplementation and verification management and control mode of production information by the digital twin system and the video stream system;

[0055] The target personnel interact immersively with the multimedia management and control platform, and inspect the digital space of the industrial production site from multiple perspectives and identities, and subscribe to information on electromechanical equipment, various personnel and industrial scenes in real time.

[0056] The present invention provides an equipment management and control system that transforms geographic space into high-precision digital space. The digital twin system is used to provide digital twin data of industrial production scenes and electromechanical equipment; the video streaming system is used to output production scene video streaming data based on orthographic projection, geometric correction, panoramic stitching, and audio fusion in real time and continuously.

[0057] The multimedia digital space system is used to integrate digital twin data and video stream data to form high-precision fused data that integrates real-time graphics, images, video, and audio, and use the information correction module to perform time and space synchronization correction on the fused data to output multimedia digital space data; the industrial control system is used to combine multimedia digital space systems based on multimedia digital space data, and use industrial control technology or systems to build a multimedia management and control platform.

[0058] The multimedia management and control platform is used to achieve remote control and fault diagnosis of electromechanical equipment at industrial production sites, digital inspection and disaster warning of industrial production scenes, and realize the mutual supplement and verification of production information by the digital twin system and video streaming system.

[0059] The equipment control system from geographic space to high-precision digital space proposed in the present invention is obviously unrealistic to achieve real-time collection of all information in the physical environment of industrial production, because the expression accuracy of the digital twin system is affected by the data acquisition or sensor system, but it can achieve the expression and real-time control of known information; and the visual and auditory effects of the video streaming system are affected by the line of sight direction, video and audio acquisition density, and the expression accuracy is also problematic, but it can supplement the information that the digital twin system cannot collect.

[0060] The present invention combines the two, and the combination of two types of spatiotemporal object information is used to make up for the loss of model accuracy and insufficient data collection in the digital twin system, and improve the information expression range and expression accuracy of digital twin objects of industrial production scenes and electromechanical equipment; make up for the insufficient acquisition density of the video stream system, and improve the information expression range and expression accuracy of video stream objects of production scenes and electromechanical equipment. Whether in the digital twin system or in the video stream system, personnel and the system can interact immersively and inspect in the digital space of the equipment control system from multiple perspectives and identities.

[0061] It not only improves the expression range and accuracy of digital twins, but can also be used in industrial production sites such as smart mines and smart factories to achieve remote and precise control and fault diagnosis of electromechanical equipment on industrial production sites; realize immersive digital inspections and disaster warnings in industrial production scenarios; realize remote and precise control and immersive inspections of smart mines, smart factories and other places, as well as production goals with few or no people. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0063] Figure 1 This is an architecture diagram of a device control system from geographic space to high-precision digital space according to an embodiment of the present invention;

[0064] Figure 2 It is a flow chart of a device control method from geographic space to high-precision digital space in an embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, are only part of the embodiments of the present invention, not all of the embodiments, and are not used to limit the present invention.

[0066] The present invention proposes an equipment management and control system from geographic space to high-precision digital space, including: a digital twin system, a video streaming system, a multimedia digital space system and an industrial control system; the digital twin system includes: an industrial production scene model, an electromechanical equipment model, a script description, and an information feedback function module, which is used to provide digital twin data of industrial production scenes and electromechanical equipment.

[0067] The video streaming system includes: panoramic camera device, image stitching, orthographic projection, geometric correction, and pixel three-dimensional geographic coordinate assignment function modules, which are used to construct vector space and real-time and continuous output of production scene video streaming data based on orthographic projection, geometric correction, panoramic stitching images and audio fusion.

[0068] The multimedia digital space system is used to fuse digital twin data and video stream data to form high-precision fused data that integrates real-time graphics, images, video, and audio, and use the information correction module to perform time and space synchronization correction on the fused data to output multimedia digital space data.

[0069] An industrial control system is used to combine multimedia digital space systems based on multimedia digital space data, and to build a multimedia management and control platform using industrial control technology or systems; the multimedia management and control platform is used to realize remote control and fault diagnosis of electromechanical equipment at industrial production sites, realize digital inspections and disaster warnings of industrial production scenes, and realize the mutual supplementation and verification of production information by the digital twin system and the video stream system.

[0070] Reference Figure 1 An exemplary architecture diagram of a device control system from geographic space to high-precision digital space is shown. Figure 1 The digital twin system includes: industrial production scene model, electromechanical equipment model, script description, information feedback (i.e. information feedback function module), and provides digital twin data (i.e. digital twin information) of industrial production scenes and electromechanical equipment.

[0071] The video stream system includes: panoramic camera device, image stitching, orthographic projection, geometric correction, and assigning three-dimensional geographic coordinates to pixels (i.e., a functional module for assigning three-dimensional geographic coordinates to pixels), and provides production scene video stream data (i.e., video stream information) based on orthographic projection, geometric correction, and panoramic stitching image and audio fusion.

[0072] The multimedia digital space system integrates digital twin information and video stream information to form high-precision fused data that integrates real-time graphics, images, video, and audio, and uses information correction (i.e., information calibration module) to perform spatiotemporal synchronization correction on the fused data to output multimedia digital space data.

[0073] The industrial control system is based on the combination of multimedia digital space data and multimedia digital space system, and uses industrial control technology or system to build a multimedia control platform. The multimedia control platform realizes remote control and fault diagnosis of electromechanical equipment on industrial production sites, realizes digital inspection and disaster warning of industrial production scenes, and realizes the mutual supplement and verification control mode of production information by the digital twin system and the video stream system ( Figure 1 For the sake of simplicity, three functions are schematically shown: equipment remote control, digital space inspection, and equipment fault diagnosis).

[0074] In the above architecture, the structural characteristics of the industrial production scene model and electromechanical equipment model in the digital twin system are the same as the structural characteristics of the scenes and equipment at the industrial site; among them, the structural characteristics include: internal and external structure, material, spatial position, geometric constraints, appearance mapping, and degrees of freedom.

[0075] The information feedback function module is used to collect structural and operating characteristic information of industrial scenes and electromechanical equipment. The script description module adopts a visual script programming method, using characteristic information to drive the update, movement and fault diagnosis of industrial scenes and electromechanical equipment, so as to realize the virtual reflecting the real, the virtual controlling the real, and the virtual and the real coexisting. That is, the digital twin system has the characteristics of the virtual reflecting the real, the virtual controlling the real, and the virtual and the real coexisting.

[0076] The video streaming system can construct vector space and real-time and continuous output of industrial production scenes and electromechanical equipment real-time orthographic projection, geometric correction, panoramic stitching images and audio fusion video streaming data. Among them, the video streaming data is a real-time and continuous data sequence, which includes: real-time image data, audio data, three-dimensional geographic coordinates assigned to pixels, and camera control instruction information; the video streaming system is configured to have the functions of audio and video feature information extraction, model training, analysis, and recognition result prompts.

[0077] The multimedia digital space system combines the two types of spatiotemporal object information, the digital twin system and the video stream system, to complement, verify and check the accuracy of information expression. Therefore, the multimedia digital space system includes both video stream information for displaying scenes and electromechanical equipment at industrial production sites, and digital twin information for displaying scenes and internal and external structural features of electromechanical equipment at industrial production sites; the information correction module is used to synchronize the spatiotemporal information of the digital twin information and the video stream information.

[0078] As explained above: since the expression accuracy of the digital twin system is affected by the data acquisition or sensor system, it is obviously unrealistic to achieve real-time collection of all information in the physical environment of industrial production, but it can achieve the expression and real-time control of known information; while the visual and auditory effects of the video streaming system are affected by the line of sight, video and audio acquisition density, and the expression accuracy is also problematic, but it can supplement the information that the digital twin system cannot collect. Therefore, the present invention creatively combines the two.

[0079] However, the integration of digital twin systems and video streaming systems is difficult to achieve. The technical difficulties that need to be overcome are mainly:

[0080] Difficulty 1: The pixels of the panoramic stitching image of the video streaming system do not accurately express the spatial position (x, y, z), and it is difficult to make them completely consistent with the spatial position of the digital twin model;

[0081] Difficulty 2: The spatial position (x, y, z) of some objects in the digital twin system is expressed accurately enough, but the environment of the production site is affected by the data acquisition or sensor system, and it is difficult to fully express its state characteristics;

[0082] Difficulty 3: The spatiotemporal synchronization of data for the same object in the video streaming system and the digital twin system is limited by computing power, network and fusion methods, making it difficult to store, render and present information synchronously.

[0083] Based on the above difficulties, the inventors have creatively proposed a method of integrating the two after extensive research. That is, the proposed method of integrating digital twin data and video stream data in a multimedia digital space system includes:

[0084] Step S1: The multimedia digital space system marks the unique numbers (n 1 ,n 2 ,n 3 ...) and coordinates ((x v1 ,y v1 , z v1 ), (x v2 ,y v2 , z v2 ), (x v3 ,y v3 , z v3 )……), store the camera point number and coordinates and other information in the database of the digital twin system;

[0085] Step S2: The multimedia digital space system calibrates the camera point position in the digital twin system and obtains the same number (n 1 ,n 2 ,n 3 ……) The coordinates of the camera point ((x s1 ,ys1 ,z s1 ),(x s2 ,y s2 ,z s2 ),(x s3 ,y s3 ,z s3 )……);

[0086] Step S3: Based on the two sets of coordinates, a mathematical model for mutual geometric correction is established to unify the two coordinate systems; if the unified coordinate system is based on the digital twin system, the three-dimensional coordinates of all pixels in the video stream system are corrected to the digital twin coordinate system; if the unified coordinate system is based on the video stream system, the three-dimensional coordinates of all pixels in the digital twin system are corrected to the video stream coordinate system;

[0087] Step S4: The digital twin system obtains the scene and structure, operation and fault, personnel behavior and environmental data of the electromechanical equipment model of the industrial production site in real time and continuously from the industrial production site. The multimedia digital space system uses the information correction module to associate the structure, operation and fault, personnel behavior and environmental data of the scene and electromechanical equipment model with the camera point number and coordinates (x, y, z), panoramic orthographic projection stitching image and audio, and establishes associations according to the time-space relationship to form fused data;

[0088] Step S5: The digital twin system queries the numbers and coordinates of all camera points, arranges the numbers in the order of inspection time or random positions, combines them into inspection routes in industrial production scenarios, and stores them in the database of the digital twin system;

[0089] Step S6: During the random inspection process of the digital twin system according to the set route or human-computer interaction, the digital twin system queries the information of the camera point closest to the observer's current position in space, constrains the digital twin scene viewpoint to the camera point, generates a combination of a 360°×360° digital twin scene and a panoramic orthographic projection stitching image, and fuses the display device monitoring information to the digital twin and video stream image of the corresponding position;

[0090] Step S7: During the inspection process along the set route, the digital twin system queries the observer's current position, sight direction, translation amount, and zoom amount, and publishes them to the video streaming system. The video streaming system subscribes to the information in real time and adjusts the observer's visual parameters to maintain the same position, sight direction, translation amount, and zoom amount as the digital twin system;

[0091] Step S8: The multimedia digital space system changes the transparency, brightness, and contrast of the panoramic orthographic projection stitching image of the video stream system and fuses the fusion data within the observation window to accurately express the scene of the industrial production site and the audio and video characteristics of the electromechanical equipment model and the operation information of the electromechanical equipment in real time and traceably;

[0092] Step S9: The multimedia digital space system accurately expresses the scenes and electromechanical equipment characteristics of the industrial production site in real time and traceably by hiding and displaying the scenes and electromechanical equipment models of the industrial production site in the digital twin system.

[0093] Through the above method, the multimedia digital space system integrates the digital twin data and the video stream data to form high-precision fused data that integrates real-time graphics, images, videos, and audio, and outputs multimedia digital space data.

[0094] The multimedia digital space system outputs multimedia digital space data, which combines and displays the scenes of industrial production sites and the structural characteristics of electromechanical equipment, operation and fault characteristics, personnel behavior, and environmental disaster identification results and linkage disposal results. Among them, combined display is a data display and expression method configured by human-computer interaction, which enables the digital twin system and the video stream system to display the scenes of industrial production sites and the information of electromechanical equipment separately or synchronously. The synchronization means: according to the observer's line of sight and the parameters of the camera, the same production scene and related information are displayed at the same time.

[0095] After the multimedia digital space system is formed, the industrial control system combines the multimedia digital space system based on the multimedia digital space data, and also uses the industrial control technology or system to build a multimedia control platform. The method of combining the multimedia digital space system with the industrial control system includes:

[0096] Step T1: Based on the multimedia digital space system, query and display the structure, operation and failure, personnel behavior and environmental data of the industrial production site scene and electromechanical equipment model;

[0097] Step T2: query the disaster warning, remote control and fault category parameters of the industrial production site, and perform disaster warning and fault diagnosis on the industrial production site;

[0098] Step T3: Based on the scenes of the industrial production site and the characteristic data of electromechanical equipment in the multimedia digital space system, establish industrial control logic to control the electromechanical equipment to move accurately according to the predetermined actions and positions;

[0099] Step T4: Display real-time images and audio of the industrial production site in the three-dimensional scene of the multimedia digital space system, and promptly trigger the shutdown and lockout of the industrial control system when danger occurs through human eye observation or video AI diagnosis;

[0100] Step T5: After the industrial control system controls the action and interlocking of the electromechanical equipment on the industrial site, the information is fed back to the digital twin system, and the system interface synchronously displays the equipment operation status characteristics.

[0101] After the industrial control system executes the control actions and interlocking linkages of the electromechanical equipment at the industrial production site, it will feed back the information to the digital twin system to enable operators to remotely immerse in the digital space and remotely control the dangerous areas and electromechanical equipment at the industrial site.

[0102] Combined with the previous explanations and illustrations, it can be seen that the fusion of the respective data of the digital twin system and the video streaming system is a mutual complement and verification of the production information, which can achieve interoperability, real-time, scalability, fidelity and closed-loop, that is, the fusion of the two has the characteristics of interoperability, real-time, scalability, fidelity and closed-loop.

[0103] The so-called interoperability refers to: the two-way, dynamic interaction and real-time connection between the digital twin system and the video streaming system and the physical objects of the industrial production scene and electromechanical equipment, and the continuous mutual complementation, verification and control of the spatiotemporal data of the digital twin system and the video streaming system;

[0104] The so-called real-time means that the digital twin data and video stream data are synchronized in time and space based on a unified clock and spatial coordinate system.

[0105] The so-called scalability means that the industrial production scenario model and electromechanical equipment model have the ability to integrate, add and replace, and the model content can be expanded.

[0106] The so-called fidelity means that while the industrial production scene model and electromechanical equipment model maintain high-fidelity simulation of the geometric structure of the physical entity, they also simulate real-time graphics, images, video, audio, state, phase and tense to ensure high similarity with the physical entity.

[0107] The so-called closed-loop means that the multimedia management and control platform is used to describe the visual model and internal mechanism of the physical entities in the industrial production site, and use the data of the physical entities to conduct analysis and reasoning, optimize process parameters and operating parameters, and realize decision-making functions, that is, use a closed-loop system for digital models and physical entities.

[0108] Therefore, whether in the digital twin system or in the video streaming system, relevant personnel can interact immersively with the equipment control system from geographic space to high-precision digital space, and patrol in the digital space of the equipment control system from geographic space to high-precision digital space with multiple perspectives, routes, degrees of freedom and identities.

[0109] Based on the above-mentioned device control system from geographic space to high-precision digital space, the present invention also proposes a device control method from geographic space to high-precision digital space, referring to Figure 2 The method comprises:

[0110] Step 201: construct a digital twin system, which includes: an industrial production scene model, an electromechanical equipment model, a script description, and an information feedback function module, and is used to provide digital twin data of industrial production scenes and electromechanical equipment;

[0111] Step 202: construct a video stream system, which includes: a panoramic camera device, image stitching, orthographic projection, geometric correction, and a pixel three-dimensional geographic coordinate assignment function module, which is used for real-time and continuous output of production scene video stream data based on orthographic projection, geometric correction, panoramic stitching images, and audio fusion;

[0112] Step 203: construct a multimedia digital space system, which is used to fuse digital twin data and video stream data to form high-precision fused data that integrates real-time graphics, images, videos, and audios, and use an information correction module to perform spatiotemporal synchronization correction on the fused data, and output multimedia digital space data. The multimedia digital space system configures a combination of digital twin data and video stream data as needed to display the scene of the industrial production site and the information of electromechanical equipment separately or synchronously;

[0113] Step 204: construct an industrial control system, the industrial control system is used to combine a multimedia digital space system based on multimedia digital space data, and to construct a multimedia control platform using industrial control technology or system; the multimedia control platform is used to realize remote control and fault diagnosis of electromechanical equipment on industrial production sites, realize digital inspection and disaster warning of industrial production scenes, and realize the mutual supplementation and verification control mode of production information by the digital twin system and the video stream system;

[0114] Step 205: The target personnel interact immersively with the multimedia management and control platform, and inspect in the digital space of the industrial production site with multiple perspectives and identities, and subscribe to information on electromechanical equipment, various personnel, and industrial scenes in real time.

[0115] The specific contents of constructing each system in the above steps 201 to 204 have been explained and illustrated in the previous text and will not be repeated here. In step 205, for the personnel who need to use the equipment control system from geographic space to high-precision digital space, that is, the target personnel, they can interact immersively with the multimedia control platform, and patrol in the digital space of the industrial production site with multiple perspectives and identities, subscribe to the information of electromechanical equipment, various personnel (such as on-site staff, troubleshooting personnel, etc.) and industrial scenes in real time, and realize remote and precise control and fault diagnosis of electromechanical equipment at the industrial production site; realize immersive digital patrol and disaster warning of industrial production scenes; realize remote and precise control and immersive patrol of smart mines, smart factories and other places.

[0116] To sum up, the equipment control system from geographic space to high-precision digital space proposed in the present invention is obviously unrealistic to achieve real-time collection of all information in the physical environment of industrial production, because the expression accuracy of the digital twin system is affected by the data acquisition or sensor system, but it can realize the expression and real-time control of known information; and the visual and auditory effects of the video stream system are affected by the line of sight direction, video and audio acquisition density, and the expression accuracy is also problematic, but it can supplement the information that the digital twin system cannot collect.

[0117] The present invention combines the two, and the combination of two types of spatiotemporal object information is used to make up for the loss of model accuracy and insufficient data collection in the digital twin system, and improve the information expression range and expression accuracy of digital twin objects of industrial production scenes and electromechanical equipment; make up for the insufficient acquisition density of the video stream system, and improve the information expression range and expression accuracy of video stream objects of production scenes and electromechanical equipment. Whether in the digital twin system or in the video stream system, personnel and the system can interact immersively and inspect in the digital space of the equipment control system from multiple perspectives and identities.

[0118] It not only improves the expression range and accuracy of digital twins, but can also be used in industrial production sites such as smart mines and smart factories to achieve remote and precise control and fault diagnosis of electromechanical equipment on industrial production sites; realize immersive digital inspections and disaster warnings in industrial production scenarios; realize remote and precise control and immersive inspections of smart mines, smart factories and other places, as well as production goals with few or no people.

[0119] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0120] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0121] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A device control system from geographic space to high-precision digital space, characterized in that: include: Digital twin systems, video streaming systems, multimedia digital space systems, and industrial control systems; The digital twin system includes: an industrial production scene model, an electromechanical equipment model, a script description, and an information feedback function module, which is used to provide digital twin data of industrial production scenes and electromechanical equipment; The video stream system includes: a panoramic camera device, image stitching, orthographic projection, geometric correction, and a pixel three-dimensional geographic coordinate assignment function module, which is used to construct a vector space and output production scene video stream data based on orthographic projection, geometric correction, and panoramic stitching images and audio fusion in real time and continuously; The multimedia digital space system is used to fuse the digital twin data and the video stream data to form high-precision fused data that integrates real-time graphics, images, videos, and audios, and use an information correction module to perform spatiotemporal synchronization correction on the fused data to output multimedia digital space data; The industrial control system is used to combine the multimedia digital space system based on the multimedia digital space data, and to construct a multimedia management and control platform using industrial control technology or system; The multimedia control platform is used to realize remote control and fault diagnosis of electromechanical equipment at industrial production sites, realize digital inspection and disaster warning of industrial production scenes, and realize the mutual supplementation and verification control mode of production information by the digital twin system and the video stream system; The method for integrating the digital twin data and the video stream data in the multimedia digital space system includes: Step S1: The multimedia digital space system marks the unique numbers (n1, n2, n3...) and coordinates ((x v1 ,y v1 , z v1 ), (x v2 ,y v2 , z v2 ), (x v3 ,y v3 , z v3 )……), and store the camera point number and coordinates and other information in the database of the digital twin system; Step S2: The multimedia digital space system calibrates the camera point position in the digital twin system, and obtains the coordinates ((x s1 ,y s1 ,z s1 ),(x s2 ,y s2 ,z s2 ),(x s3 ,y s3 ,z s3 )……); Step S3: Based on the two sets of coordinates, a mathematical model for mutual geometric correction is established to achieve the unification of the two coordinate systems; if the unified coordinate system is based on the digital twin system, the three-dimensional coordinates of all pixels in the video stream system are corrected to the digital twin coordinate system; if the unified coordinate system is based on the video stream system, the three-dimensional coordinates of all pixels in the digital twin system are corrected to the video stream coordinate system; Step S4: The digital twin system obtains the scene of the industrial production site and the structure, operation and failure, personnel behavior and environmental data of the electromechanical equipment model in real time and continuously from the industrial production site. The media digital space system uses the information correction module to associate the structure, operation and failure, personnel behavior and environmental data of the scene and electromechanical equipment model with the camera point number and coordinates (x, y, z), panoramic orthographic projection stitching image and audio, and establish associations according to the time and space relationship to form fused data; Step S5: The digital twin system queries the numbers and coordinates of all camera points, arranges the numbers in the order of inspection time or random positions, combines them into inspection routes in industrial production scenarios, and stores them in the database of the digital twin system; Step S6: The digital twin system follows the set route or human-computer interaction During the random inspection process, the camera point information closest to the observer's current spatial position is queried, and the digital twin scene viewpoint is constrained to the camera point to generate a combination of 360°×360° digital twin scene and panoramic orthographic projection stitching image, and the display device monitoring information is integrated into the digital twin and video stream image of the corresponding position; Step S7: During the inspection process of the digital twin system according to the set route, the observer's current position, line of sight direction, translation amount, and zoom amount are queried, and published to the video stream system. The video stream system subscribes to the information in real time and adjusts the observer's visual parameters to keep the same as the digital twin system position, sight direction, translation and zoom amount; Step S8: The multimedia digital space system changes the transparency, brightness and contrast of the panoramic orthographic projection stitching image of the video stream system, and fuses the fused data within the observation window, so as to accurately express the scene of the industrial production site and the audio and video characteristics of the electromechanical equipment model and the operation information of the electromechanical equipment in real time and traceably; Step S9: The multimedia digital space system hides and displays the scene of the industrial production site and the electromechanical equipment model in the digital twin system, so as to accurately express the scene of the industrial production site and the characteristics of the electromechanical equipment in real time and traceably.

2. The equipment control system according to claim 1, characterized in that: The structural features of the industrial production scene model and the electromechanical equipment model are the same as those of the scenes and equipment at the industrial site; The structural features include: internal and external structure, material, spatial position, geometric constraints, appearance mapping, and degrees of freedom; The information feedback function module is used to collect industrial scene and electromechanical equipment structure and operation characteristic information; The script description module adopts the visual script programming method and uses feature information to drive industrial scenes and electromechanical equipment to update, move and diagnose faults, so as to achieve virtual reflection of reality, virtual control of reality, and symbiosis of virtual and real.

3. The equipment control system according to claim 1, characterized in that: The video stream system constructs a vector space and outputs real-time and continuous video stream data of real-time orthographic projection, geometric correction, panoramic stitching images and audio fusion of the industrial production scene and the electromechanical equipment; The video stream data is a real-time, continuous data sequence, which includes: real-time image data, audio data, three-dimensional geographic coordinates assigned to pixels, and camera control instruction information; The video streaming system is configured to have the functions of audio and video feature information extraction, model training, analysis, and recognition result prompting.

4. The equipment control system according to claim 1, characterized in that: The digital twin system is used to collect known information of the physical environment of industrial production in real time, so as to express and control the known information in real time; The video stream system is used to supplement the information that cannot be collected by the digital twin system; The two types of spatiotemporal object information of the digital twin system and the video stream system are combined to complement, verify and check the expression accuracy of the information.

5. The equipment control system according to claim 1, characterized in that: The multimedia digital space system outputs multimedia digital space data, which combines and displays the scenes of industrial production sites and the structural characteristics, operation and fault characteristics of electromechanical equipment, personnel behavior, and environmental disaster identification results and linkage disposal results; Among them, the combined display is a data display and expression method of human-computer interaction configuration, so that the digital twin system and the video stream system can display the scenes and information of electromechanical equipment at the industrial production site separately or synchronously. The synchronization means: according to the direction, position, translation and zoom parameters of the observer's line of sight, the same production scene and related information are displayed at the same time.

6. The equipment control system according to claim 1, characterized in that: The method for combining the industrial control system with the multimedia digital space system includes: Step T1: Based on the multimedia digital space system, query and display the structure, operation and failure, personnel behavior and environmental data of the industrial production site scene and electromechanical equipment model; Step T2: query the disaster warning, remote control and fault category parameters of the industrial production site, and perform disaster warning and fault diagnosis on the industrial production site; Step T3: Based on the scenes of the industrial production site and the characteristic data of the electromechanical equipment of the multimedia digital space system, an industrial control logic is established to control the electromechanical equipment to move accurately according to the predetermined actions and positions; Step T4: Displaying real-time images and audio of the industrial production site in the three-dimensional scene of the multimedia digital space system, and timely triggering the shutdown and locking of the industrial control system when danger occurs through human eye observation or video AI diagnosis; Step T5: After the industrial control system controls the action and locking linkage of the electromechanical equipment on the industrial site, the information is fed back to the digital twin system, and the system interface synchronously displays the equipment operation status characteristics.

7. The equipment control system according to claim 6, characterized in that: After the industrial control system executes the control actions and interlocking linkage of the electromechanical equipment at the industrial production site, it feeds back the information to the digital twin system to enable operators to remotely immerse in digital space inspection and remote control of dangerous areas and electromechanical equipment at the industrial site.

8. A device control method from geographic space to high-precision digital space, characterized in that: include: Constructing a digital twin system, the digital twin system comprising: an industrial production scene model, an electromechanical equipment model, a script description, and an information feedback function module, for providing digital twin data of industrial production scenes and electromechanical equipment; Constructing a video streaming system, the video streaming system comprising: a panoramic camera device, an image stitching, an orthographic projection, a geometric correction, and a pixel three-dimensional geographic coordinate assignment function module, for constructing a vector space and real-time and continuous output of production scene video streaming data based on orthographic projection, geometric correction, and panoramic stitching images and audio fusion; Construct a multimedia digital space system, the multimedia digital space system is used to fuse the digital twin data and the video stream data to form high-precision fused data that integrates real-time graphics, images, videos, and audios, and use an information correction module to perform spatiotemporal synchronization correction on the fused data to output multimedia digital space data. The multimedia digital space system is configured to combine and display the digital twin data and the video stream data as needed, and the scenes of the industrial production site and the information of the electromechanical equipment are displayed separately or synchronously; wherein the method for fusing the digital twin data and the video stream data in the multimedia digital space system comprises: step S1: the multimedia digital space system calibrates the unique numbers (n1, n2, n3...) and coordinates ((x v1 ,y v1 , z v1 ), (x v2 ,y v2 , z v2 ), (x v3 ,y v3 , z v3 )……), and store the camera point number and coordinates and other information in the database of the digital twin system; Step S2: The multimedia digital space system calibrates the camera point position in the digital twin system, and obtains the coordinates ((x s1 ,y s1 ,z s1 ),(x s2 ,y s2 ,z s2 ),(x s3 ,y s3 ,z s3 )……); Step S3: Based on the two sets of coordinates, a mathematical model for mutual geometric correction is established to achieve the unification of the two coordinate systems; if the unified coordinate system is based on the digital twin system, the three-dimensional coordinates of all pixels in the video stream system are corrected to the digital twin coordinate system; if the unified coordinate system is based on the video stream system, the three-dimensional coordinates of all pixels in the digital twin system are corrected to the video stream coordinate system; Step S4: The digital twin system obtains the scene of the industrial production site and the structure, operation and failure, personnel behavior and environmental data of the electromechanical equipment model in real time and continuously from the industrial production site. The media digital space system uses the information correction module to associate the structure, operation and failure, personnel behavior and environmental data of the scene and electromechanical equipment model with the camera point number and coordinates (x, y, z), panoramic orthographic projection stitching image and audio, and establish associations according to the time and space relationship to form fused data; Step S5: The digital twin system queries the numbers and coordinates of all camera points, arranges the numbers in the order of inspection time or random positions, combines them into inspection routes in industrial production scenarios, and stores them in the database of the digital twin system; Step S6: The digital twin system follows the set route or human-computer interaction During the random inspection process, the camera point information closest to the observer's current spatial position is queried, and the digital twin scene viewpoint is constrained to the camera point to generate a combination of 360°×360° digital twin scene and panoramic orthographic projection stitching image, and the display device monitoring information is integrated into the digital twin and video stream image of the corresponding position; Step S7: During the inspection process of the digital twin system according to the set route, the observer's current position, line of sight direction, translation amount, and zoom amount are queried, and published to the video stream system. The video stream system subscribes to the information in real time and adjusts the observer's visual parameters to keep the same as the digital twin system The position, sight direction, translation amount and zoom amount of the industrial production site are accurately expressed in real time and traceably by changing the transparency, brightness and contrast of the panoramic orthographic projection stitching image of the video stream system and fusing the fused data within the observation window; Step S9: The multimedia digital space system accurately expresses the scene and electromechanical equipment characteristics of the industrial production site in real time and traceably by hiding and showing the scene and electromechanical equipment model of the industrial production site in the digital twin system; Constructing an industrial control system, the industrial control system is used to combine the multimedia digital space system based on the multimedia digital space data, and to construct a multimedia management and control platform using industrial control technology or system; the multimedia management and control platform is used to realize remote control and fault diagnosis of electromechanical equipment at industrial production sites, realize digital inspection and disaster warning of industrial production scenes, and realize the mutual supplementation and verification management and control mode of production information by the digital twin system and the video stream system; The target personnel interact immersively with the multimedia management and control platform, and inspect the digital space of the industrial production site from multiple perspectives and identities, and subscribe to information on electromechanical equipment, various personnel and industrial scenes in real time.

Images