Panoramic monitoring method and device for industrial equipment, electronic equipment and storage medium

Abstract

The invention provides a panoramic monitoring method and device for industrial equipment, electronic equipment and a storage medium, and the method comprises the steps: obtaining video data shot by a plurality of cameras corresponding to each piece of equipment in a preset region in real time, and enabling the shooting angles of the plurality of cameras corresponding to the equipment to cover each angle of the equipment; for each device, splicing the plurality of video data corresponding to the device to obtain a panoramic video image; performing 3D modeling on the equipment based on the panoramic video image to obtain a 3D equipment image; and the 3D equipment image is displayed on the monitoring display interface, so that the technical problem of low monitoring efficiency of the industrial equipment in the prior art is solved, and the monitoring accuracy of the industrial equipment is ensured.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and particularly to a panoramic monitoring method, device, electronic device, and storage medium for industrial equipment. Background Art

[0002] In industrial production, the stable operation of industrial equipment is crucial. Currently, the monitoring of industrial equipment mainly relies on manual inspections. During daily operation, manual workers need to regularly check industrial equipment to detect faults such as leaks; and when a fault occurs in industrial equipment, manual workers also need to go to the site for a detailed inspection. Since there are multiple industrial equipment in the production site and the industrial equipment is relatively large, it is time-consuming and laborious to monitor the industrial equipment in the production site manually, thus resulting in low monitoring efficiency of the equipment. Summary of the Invention

[0003] In view of this, the purpose of this application is to propose a panoramic monitoring method, device, electronic device, and storage medium for industrial equipment to overcome all or part of the deficiencies in the prior art.

[0004] Based on the above purpose, this application provides a panoramic monitoring method for industrial equipment, including: obtaining video data captured by multiple cameras corresponding to each device within a predetermined area in real time, where the shooting angles of the multiple cameras corresponding to the device cover all angles of the device; for each device, performing splicing processing on the multiple video data corresponding to the device to obtain a panoramic video image; based on the panoramic video image, performing 3D modeling on the device to obtain a 3D device image; and displaying the 3D device image on a monitoring display interface.

[0005] Optionally, the performing splicing processing on the multiple video data corresponding to the device to obtain a panoramic video image includes: for each video data corresponding to the device, performing feature extraction on the video data to obtain multiple device features corresponding to the video data; for each device feature, searching for a device feature having an association relationship with the device feature among all device features corresponding to the device to form a device feature group; and based on all device feature groups, using a predetermined image splicing technique to splice all device features to obtain the panoramic video image.

[0006] Optionally, the performing 3D modeling on the device based on the panoramic video image to obtain a 3D device image includes: searching for a device three-dimensional model corresponding to the device in a pre-constructed device three-dimensional model library; and mapping the detailed information in the panoramic video image to the device three-dimensional model to obtain the 3D device image.

[0007] Optionally, after the 3D device image is displayed on the monitoring display interface, the method includes: in response to receiving an operation instruction for the 3D device image, controlling the 3D device image to be displayed according to the operation instruction.

[0008] Optionally, after the 3D device image is displayed on the monitoring display interface, the method further includes: at the moment when the 3D device image is displayed on the monitoring display interface, associating and archiving the 3D device image, as well as a plurality of video data and panoramic video images corresponding to the 3D device image, to a pre-constructed database.

[0009] Optionally, the method further includes: in response to determining that the device has a fault, obtaining the fault moment when the device has the fault; respectively searching in the database for the 3D device image corresponding to each moment within a predetermined range of time difference from the fault moment, as well as a plurality of video data and panoramic video images corresponding to each 3D device image.

[0010] Optionally, the method further includes: in response to receiving a camera parameter adjustment instruction, adjusting the parameters of the camera corresponding to the camera parameter adjustment instruction.

[0011] Based on the same inventive concept, the present application further provides a panoramic monitoring device for industrial equipment, including: an acquisition module configured to acquire in real time video data captured by a plurality of cameras corresponding to each device within a predetermined area, wherein the shooting angles of the plurality of cameras corresponding to the device cover all angles of the device; a splicing processing module configured to perform splicing processing on the plurality of video data corresponding to each device to obtain a panoramic video image; a modeling module configured to perform 3D modeling on the device based on the panoramic video image to obtain a 3D device image; and a display module configured to display the 3D device image on a monitoring display interface.

[0012] Based on the same inventive concept, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable by the processor, where the processor implements the method as described above when executing the computer program.

[0013] Based on the same inventive concept, the present application further provides a non-transitory computer-readable storage medium, where the non-transitory computer-readable storage medium stores computer instructions for causing a computer to execute the method as described above.

[0014] As can be seen from the above, the panoramic monitoring method, device, electronic device and storage medium of industrial equipment provided by the present application, the method includes obtaining video data captured by a plurality of cameras corresponding to each device in a predetermined area in real time, wherein the shooting angles of the plurality of cameras corresponding to the device cover all angles of the device, ensuring comprehensive and real-time acquisition of video data of each device at different angles and eliminating monitoring blind spots for each device. For each device, the plurality of video data corresponding to the device are spliced to obtain a panoramic video image, so as to achieve the purpose of plane restoration of the device in all dimensions. Based on the panoramic video image, 3D modeling is performed on the device to obtain a 3D device image, so as to achieve the purpose of accurate three-dimensional restoration of the device. The 3D device image is displayed on the monitoring display interface, improving the timeliness and accuracy of device monitoring. Description of the Drawings

[0015] In order to more clearly illustrate the technical solutions in the present application or related technologies, the following will briefly introduce the drawings required for use in the embodiments or related technology descriptions. Obviously, the drawings in the following description are only embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 It is a schematic flowchart of the panoramic monitoring method of industrial equipment according to an embodiment of the present application;

[0017] Figure 2 It is a schematic diagram of camera installation according to an embodiment of the present application;

[0018] Figure 3 It is a schematic structural diagram of the panoramic monitoring device of industrial equipment according to an embodiment of the present application;

[0019] Figure 4 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present application. Detailed Embodiments

[0020] In order to make the purpose, technical solutions and advantages of the present application clearer, the following will further describe the present application in detail with reference to specific embodiments and the accompanying drawings.

[0021] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of this application should have the ordinary meanings understood by those with ordinary skills in the field to which this application belongs. The "first", "second" and similar terms used in the embodiments of this application do not denote any order, quantity or importance, but are only used to distinguish different components. Words such as "including" or "comprising" mean that the elements or objects appearing before this word cover the elements or objects listed after this word and their equivalents, without excluding other elements or objects. Words such as "connected" or "coupled" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Upper", "lower", "left", "right", etc. are only used to represent relative positional relationships, and when the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0022] As described in the background art section, in industrial production, the stable operation of industrial equipment is crucial. Currently, for the monitoring of industrial equipment, it mainly relies on manual inspections. During daily operation, workers need to regularly check industrial equipment to detect faults such as leaks; and when a fault occurs in industrial equipment, workers also need to go to the site for detailed inspections. Since there are multiple industrial equipment at the production site and the industrial equipment is relatively large, it is time-consuming and laborious to monitor the industrial equipment at the production site manually, thus resulting in low monitoring efficiency of the equipment.

[0023] In addition, the method of manual on-site inspection and monitoring also has the following drawbacks: on the one hand, when manual monitoring detects a fault in the equipment, it is often discovered after the fault has occurred, lacking visual records of the fault moment, resulting in a lack of sufficient intuitive information support during fault restoration and cause analysis, making it more difficult to troubleshoot and repair faults. On the other hand, for some special equipment, such as high-pressure reactors, high-temperature furnaces, etc., their working environments have risk factors such as high noise, high pressure, high temperature, and strong radiation. Although manual on-site inspection and monitoring can detect problems more carefully, this is at the cost of the health and safety of the inspection personnel. Being in such an environment for a long time, the inspection personnel are extremely vulnerable to injury.

[0024] Although some existing monitoring means use multiple fixed cameras for monitoring, these cameras often simply take partial pictures of the equipment, unable to form a full-dimensional and non-blind-spot visual monitoring effect, and it is also difficult to achieve real-time 3D (Three-Dimensional) display of the equipment at the terminal, unable to meet the requirements of remote monitoring of the equipment and querying the historical operating status, thus resulting in low accuracy of equipment monitoring.

[0025] In view of this, the embodiments of this application propose a panoramic monitoring method for industrial equipment, refer toFigure 1 , including the following steps:

[0026] Step 101, obtain in real time the video data captured by multiple cameras corresponding to each device within a predetermined area, wherein the shooting angles of the multiple cameras corresponding to the device cover all angles of the device.

[0027] In this step, the stable operation of industrial equipment is crucial for industrial production. In the prior art, it is usually necessary to rely on manual monitoring of industrial equipment at the production site regularly. However, the efficiency of the above monitoring is low. In this application, the video data captured by multiple cameras corresponding to each device within a predetermined area is obtained in real time. Among them, the predetermined area can be an industrial production site, and the device is an industrial equipment to be monitored. For example, the predetermined key industrial equipment at the industrial production site. As Figure 2 shown, according to the shape, size and structural characteristics of the observed device, multiple high-definition cameras are installed at different positions and angles around the device. The number of installed cameras depends on the device to ensure that all cameras cover all angles of the device. Exemplarily, for a cylindrical chemical reactor, cameras are installed at different heights and angles on its top and side to ensure that the entire surface and all connection parts of the reactor can be photographed. The shooting angles of the multiple cameras corresponding to the device cover all angles of the device. These cameras cover all surfaces and key parts of the device, ensuring no monitoring dead angles to obtain video data from different perspectives of the device.

[0028] Each camera captures the device in real time according to the set frame rate and resolution. After the cameras are installed, connect the cameras to the data processing system through network cables or wireless transmission modules, and ensure the stability and reliability of the transmission line. The cameras transmit the captured video data to the data processing system in this application quickly and stably through wired or wireless transmission. This application is applied to the data processing system. The video data transmission status is monitored in real time. In case of network failure or signal interference, the data processing system automatically retransmits or switches the transmission channel to ensure the integrity and real-time nature of the video data. To ensure the real-time nature of the video data, an efficient data compression algorithm can be adopted. For example, use high-efficiency data compression algorithms such as H.265 (High Efficiency Video Coding) or H.264 (Advanced Video Coding) for compression to reduce the amount of video data transmitted, thereby improving the transmission efficiency of the video data. By obtaining in real time the video data captured by multiple cameras corresponding to each device within a predetermined area, it is ensured that the video data of each device at different angles is obtained comprehensively and in real time, eliminating the monitoring dead angles for each device.

[0029] Step 102: For each device, splice the multiple video data corresponding to the device to obtain a panoramic video image.

[0030] In this step, for each device, since the multiple video data of the device reflect the video data of the device at various angles, the multiple video data corresponding to the device are spliced to seamlessly fit the video data at different angles together to obtain a panoramic video image. Through the panoramic video image, the purpose of performing a full-dimensional planar restoration of the device is achieved.

[0031] Step 103: Based on the panoramic video image, perform 3D modeling on the device to obtain a 3D device image.

[0032] In this step, in order to endow the panoramic video image with depth information and spatial dimensions, 3D modeling is performed on the device based on the panoramic video image to obtain a 3D device image. Through the 3D device image, the purpose of performing an accurate three-dimensional restoration of the device is achieved.

[0033] Step 104: Display the 3D device image on the monitoring display interface.

[0034] In this step, when the 3D device image is displayed on the monitoring display interface, since the 3D device image can accurately perform real-time three-dimensional restoration of the device and perform real-time 3D display of the device, workers can know the current state of the device through the monitoring display interface without going to the production site. On the one hand, the operation status of the device can be intuitively viewed from different angles, and the device can be comprehensively monitored without going to the site, which is convenient for remote inspection, increasing the convenience of device monitoring and improving the timeliness and accuracy of device monitoring. On the other hand, it also avoids manual on-site inspections in dangerous environments such as high noise, high pressure, and high temperature, protecting the health and safety of the inspection personnel.

[0035] The monitoring display interface can determine the terminal device carried by the monitoring display interface according to a predetermined requirement. For example, the terminal device is a computer, a mobile phone, a tablet, etc. The monitoring software in the terminal device provides a graphical operation interface. It should be noted that the data processing system in this application includes the terminal device. The data processing system can be the terminal device, or the terminal device has a connection relationship with the data processing system. The terminal device is a subsystem in the data processing system, and the data processing system controls the 3D device image to be displayed on the monitoring display interface of the terminal device. A dedicated monitoring software is installed on the terminal device, and users can view the real-time 3D monitoring screen of the device and query the historical operation status on the monitoring software interface, realizing full-dimensional and non-blind-spot visual monitoring of the device.

[0036] Exemplarily, taking the monitoring of a high-pressure reactor in a chemical plant as an example, 8 high-definition cameras are installed around the reactor, located at different positions on the top, side, and bottom of the reactor. During the operation of the reactor, the cameras collect video data in real time and transmit it to the data processing system. After video fitting and 3D modeling by the data processing system, a real-time 3D monitoring screen of the reactor is presented on the computer terminal in the monitoring room. The operator can clearly see the operation conditions of various parts of the reactor, including the opening and closing states of valves, the material flow in pipelines, etc.

[0037] Through the above solution, video data captured by multiple cameras corresponding to each device within a predetermined area is obtained in real time. Among them, the shooting angles of the multiple cameras corresponding to the device cover all angles of the device, ensuring comprehensive and real-time acquisition of video data of each device at different angles and eliminating blind spots in the monitoring of each device. For each device, the multiple video data corresponding to the device are subjected to stitching processing to obtain a panoramic video image, achieving the purpose of planar restoration of the device in all dimensions. Based on the panoramic video image, 3D modeling is performed on the device to obtain a 3D device image, achieving the purpose of accurate three-dimensional restoration of the device. The 3D device image is displayed on the monitoring display interface, improving the timeliness and accuracy of device monitoring.

[0038] In some embodiments, the step of performing stitching processing on the multiple video data corresponding to the device to obtain a panoramic video image includes: for each video data corresponding to the device, performing feature extraction on the video data to obtain multiple device features corresponding to the video data; for each device feature, searching for device features having an association relationship with the device feature among all the device features corresponding to the device to form a device feature group; based on all the device feature groups, using a predetermined image stitching technique to stitch all the device features to obtain the panoramic video image.

[0039] In this embodiment, an advanced video fitting algorithm is used to analyze and process the video data obtained from multiple cameras. For each piece of video data corresponding to the device, feature extraction is performed on the video data to obtain multiple device features corresponding to the video data. By performing feature extraction on the video data, the features of the device in the video data are accurately extracted, excluding the interference of irrelevant data. The device features correspond to different angles of the device, and there is an association relationship between adjacent angles. Furthermore, there is an association relationship between the device features corresponding to adjacent angles. Therefore, for each device feature, device features having an association relationship with the device feature are searched for among all the device features corresponding to the device to form a device feature group. By comparing the device features of different video data, an association relationship between the device features of the same device is established. Based on all the device feature groups, all the device features are stitched using a predetermined image stitching technique to obtain a panoramic video image. Among them, the predetermined image stitching technique includes, but is not limited to, the SIFT (Scale-Invariant Feature Transform) algorithm or the SURF (Speeded Up Robust Features) algorithm. Since the multiple pieces of video data corresponding to the device are video data of various angles of the device and there is no dead angle for photographing the device, the multiple device features corresponding to the multiple pieces of video data of the device can be accurately stitched into a panoramic video image of the device, achieving the purpose of restoring the device in all dimensions.

[0040] In some embodiments, performing 3D modeling on the device based on the panoramic video image to obtain a 3D device image includes: searching for a device three-dimensional model corresponding to the device in a pre-constructed device three-dimensional model library; and mapping the detailed information in the panoramic video image to the device three-dimensional model to obtain the 3D device image.

[0041] In this embodiment, there are various devices in the production site. Using 3D reconstruction technology, a device three-dimensional model corresponding to each device can be pre-constructed and stored in the device three-dimensional model library. Among them, the device three-dimensional model refers to a three-dimensional visual representation of the device created using computer technology. Such a model can display the appearance, structure, functional components of the device, and their relative positional relationships. Searching for a device three-dimensional model corresponding to the device in the device three-dimensional model library. At this time, the device three-dimensional model corresponding to the device cannot accurately restore the device. The detailed information of the device can be reflected in the panoramic video image. For example, the detailed information includes the color and texture of the device, etc. Therefore, the detailed information in the panoramic video image is mapped to the device three-dimensional model to obtain a 3D device image, using the 3D form to restore the device, achieving the purpose of accurately restoring the device.

[0042] In some embodiments, after the 3D device image is displayed on the monitoring display interface, the method includes: in response to receiving an operation instruction for the 3D device image, controlling the 3D device image to be displayed according to the operation instruction.

[0043] In this embodiment, the user can issue an operation instruction for the 3D device image through the monitoring display interface by clicking with a mouse or in a touch screen manner. Among them, the operation instruction includes, but is not limited to, an image rotation instruction or an image scaling instruction. When an operation instruction for the 3D device image is received, controlling the 3D device image to be displayed according to the operation instruction can enable the running state of the device to be viewed from different angles, so as to achieve full-dimensional and non-blind-spot visual monitoring of the device, and the device state can be comprehensively understood without going to the site, improving the convenience and timeliness of device monitoring.

[0044] In some embodiments, after the 3D device image is displayed on the monitoring display interface, the method further includes: at the moment when the 3D device image is displayed on the monitoring display interface, associating and archiving the 3D device image, as well as a plurality of video data and panoramic video images corresponding to the 3D device image, to a pre-constructed database.

[0045] In this embodiment, since the 3D device image is displayed in real time, the user may have a query need for the already displayed 3D device image. Therefore, at the moment when the 3D device image is displayed on the monitoring interface, the 3D device image, as well as a plurality of video data and panoramic videos corresponding to the 3D device image, are convenient for meeting the subsequent query needs of the user. Exemplarily, the display moment is x year x month x day x minute x second, and the display moment is specific to the second, which is convenient for the user to perform accurate queries. Since the data to be stored in this application is relatively large, a large-capacity storage device is required to store the database. For example, the database is stored in a disk array or cloud storage, achieving the purpose of backing up the data corresponding to the device. It should be noted that the data in the database can be periodically cleared according to a predetermined requirement. Storing the data corresponding to the device for a long time is convenient for the user to query the historical running state of the device at any time, which helps to analyze the running trend of the device, discover potential fault hazards in advance, and realize the preventive maintenance of the device. During the storage process, a reasonable data storage structure and backup strategy are adopted to ensure the security and recoverability of the data, so as to query the historical running state of the device later.

[0046] In some embodiments, the method further includes: in response to determining that the device has a fault, obtaining the fault moment when the device has the fault; respectively searching in the database for the 3D device image corresponding to each moment within a predetermined range of time difference from the fault moment, as well as a plurality of video data and panoramic video images corresponding to each 3D device image.

[0047] In this embodiment, in the case of a device failure, the failure moment when the device fails is obtained. Since the failure may not occur instantaneously, it is also necessary to view the recent data of the device stored in the data to accurately analyze the cause of the failure. Therefore, the failure moment and the 3D device images corresponding to each moment within a predetermined range of time difference from the failure moment are respectively searched in the database, and multiple video data and panoramic video data corresponding to each 3D device image are also searched. The moments with a time difference within a predetermined range from the failure moment are the moments close to the failure moment, and the time difference between each moment and the failure moment belongs to the predetermined range. Among them, the predetermined range can be determined according to historical experience. The above data completely records the operating state of the device, provides rich visual materials for failure restoration and cause analysis, and greatly improves the efficiency of troubleshooting.

[0048] It should be noted that in the case of a user's query requirement, the user can also input the query time range through the terminal device, retrieve the corresponding data from the database, and display it in 3D on the monitoring display interface to facilitate the user to view the operating conditions of the device within a specific time period.

[0049] Exemplarily, when an abnormal situation occurs in the reactor, the operator can query the historical data to view the operating state of the reactor before the failure, and combine the 3D display function to observe the details of the device from different angles to quickly analyze the cause of the failure. For example, in a failure where the temperature of the reactor rises abnormally, the operator discovers through viewing the historical 3D monitoring data that it is due to the blockage of a certain feed valve, resulting in abnormal material flow and then causing the temperature to rise. In this way, the troubleshooting time is greatly shortened, the production efficiency is improved, and the production safety is ensured.

[0050] In some embodiments, the method further includes: in response to receiving a camera parameter adjustment instruction, adjusting the parameters of the camera corresponding to the camera parameter adjustment instruction.

[0051] In this embodiment, the high-definition camera collects video at a pre-set frame rate and resolution. Exemplarily, the frame rate range is 15 - 60 frames per second, and the resolution is not lower than 720P. For example, the frame rate is 25 frames per second and the resolution is 1080P, ensuring the clarity and accuracy of the video data. In the case of a user's predetermined requirement, the user can also adjust the parameters of at least one camera among all the cameras. When receiving a camera parameter adjustment instruction, the parameters of the camera corresponding to the camera parameter adjustment instruction are adjusted to achieve the purpose of flexibly adjusting the camera parameters, and thus the amount of video data is also flexibly adjusted.

[0052] It should be noted that in response to receiving a system parameter adjustment instruction, the system parameters corresponding to the system parameter instruction are adjusted. For example, the system parameter is the storage duration of data.

[0053] In another embodiment provided by the present application, in response to detecting a fault in the device, a warning message is generated and sent. When a fault in the device is detected, a warning message is automatically generated and sent so that the monitoring personnel can promptly discover the fault in the device. Exemplarily, in response to determining that the similarity between the image of one angle corresponding to the 3D device image and one of the predetermined fault images corresponding to this image is greater than a predetermined similarity, it is determined that a fault in the device is detected. The 3D device image includes images of multiple angles of the device. If the similarity between the image of one angle of the 3D device image and one of the pre-stored predetermined fault images corresponding to this image is greater than the predetermined similarity, it indicates that there is a fault in the device, that is, a fault in the device is detected, facilitating the timely discovery of device faults and improving the monitoring efficiency of industrial devices. It should be noted that each angle of the image corresponding to the 3D device image corresponds to multiple predetermined fault images.

[0054] It should be noted that the method of the embodiments of the present application can be executed by a single device, such as a computer or a server, etc. The method of this embodiment can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In such a distributed scenario, one of the multiple devices can only execute one or more steps of the method of the embodiments of the present application, and these multiple devices will interact with each other to complete the described method.

[0055] It should be noted that some embodiments of the present application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be executed in a different order from that in the above embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired result. In certain embodiments, multi-tasking and parallel processing are also possible or may be advantageous.

[0056] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a panoramic monitoring device for industrial equipment.

[0057] Reference Figure 3 , the panoramic monitoring device for industrial equipment includes:

[0058] An acquisition module 10, configured to acquire in real time video data captured by multiple cameras corresponding to each device within a predetermined area, wherein the shooting angles of the multiple cameras corresponding to the device cover all angles of the device.

[0059] The stitching processing module 20 is configured to perform stitching processing on multiple video data corresponding to each device to obtain a panoramic video image.

[0060] The modeling module 30 is configured to perform 3D modeling on the device based on the panoramic video image to obtain a 3D device image.

[0061] The display module 40 is configured to display the 3D device image on a monitoring display interface.

[0062] Through the above device, multiple video data captured by multiple cameras corresponding to each device within a predetermined area are obtained in real time. Among them, the shooting angles of the multiple cameras corresponding to the device cover all angles of the device, ensuring comprehensive and real-time acquisition of video data of each device at different angles and eliminating the monitoring blind spots of each device. For each device, the multiple video data corresponding to the device are stitched to obtain a panoramic video image, achieving the purpose of fully dimensional planar restoration of the device. Based on the panoramic video image, 3D modeling is performed on the device to obtain a 3D device image, achieving the purpose of accurate three-dimensional restoration of the device. Displaying the 3D device image on the monitoring display interface improves the timeliness and accuracy of device monitoring.

[0063] In some embodiments, the stitching processing module 20 is further configured to extract features from each video data corresponding to the device to obtain multiple device features corresponding to the video data; for each device feature, search for device features having an association relationship with the device feature among all the device features corresponding to the device to form a device feature group; based on all the device feature groups, use a predetermined image stitching technique to stitch all the device features to obtain the panoramic video image.

[0064] In some embodiments, the modeling module 30 is further configured to search for a device three-dimensional model corresponding to the device in a pre-constructed device three-dimensional model library; map the detailed information in the panoramic video image to the device three-dimensional model to obtain the 3D device image.

[0065] In some embodiments, a control module is further included, and the control module is configured to, after displaying the 3D device image on the monitoring display interface, in response to receiving an operation instruction for the 3D device image, control the 3D device image to be displayed according to the operation instruction.

[0066] In some embodiments, an archiving module is further included, and the archiving module is configured to, after the 3D device image is displayed on the monitoring display interface, associate and archive the 3D device image, as well as a plurality of video data and panoramic video images corresponding to the 3D device image, to a pre-constructed database according to the time when the 3D device image is displayed on the monitoring display interface.

[0067] In some embodiments, a searching module is further included, and the searching module is configured to, in response to determining that the device has a fault, obtain the fault time when the device has the fault; respectively search in the database for the 3D device images corresponding to the fault time and each time within a predetermined range of time difference from the fault time, as well as a plurality of video data and panoramic video images corresponding to each 3D device image.

[0068] In some embodiments, an adjusting module is further included, and the adjusting module is configured to, in response to receiving a camera parameter adjustment instruction, adjust the parameters of the camera corresponding to the camera parameter adjustment instruction.

[0069] For convenience of description, when describing the above device, various modules are separately described according to their functions. Of course, when implementing the present application, the functions of each module can be implemented in one or more software and / or hardware.

[0070] The device in the above embodiment is used to implement the panoramic monitoring method of the corresponding industrial device in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0071] Based on the same inventive concept, corresponding to the method in any of the above embodiments, the present application further provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the panoramic monitoring method of the industrial device as described in any of the above embodiments.

[0072] Figure 4 FIG. shows a more specific schematic diagram of the hardware structure of the electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. Among them, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other inside the device through the bus 1050.

[0073] The processor 1010 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0074] The memory 1020 can be implemented in the form of a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs. When implementing the technical solutions provided in the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1020 and are called and executed by the processor 1010.

[0075] The input / output interface 1030 is used to connect to an input / output module to achieve information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Among them, the input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0076] The communication interface 1040 is used to connect to a communication module (not shown in the figure) to achieve communication interaction between this device and other devices. Among them, the communication module can achieve communication through a wired method (such as USB, network cable, etc.) or can also achieve communication through a wireless method (such as a mobile network, WIFI, Bluetooth, etc.).

[0077] The bus 1050 includes a path for transmitting information between various components of the device (such as the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040).

[0078] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in the specific implementation process, this device may also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device may also only include the components necessary to implement the solutions of the embodiments of this specification and do not necessarily include all the components shown in the figure.

[0079] The electronic device of the above embodiment is used to implement the panoramic monitoring method of the corresponding industrial device in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0080] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the panoramic monitoring method of the industrial device as described in any of the foregoing embodiments.

[0081] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. The information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0082] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to execute the panoramic monitoring method of the industrial device as described in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0083] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a computer program product including computer program instructions, which when running on a computer, cause the computer to execute the panoramic monitoring method of the industrial device as described in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be elaborated here.

[0084] It should be noted that the embodiments of the present application can be further described in the following ways:

[0085] It can be understood that before using the technical solutions of the various embodiments in the present disclosure, the types, usage scopes, usage scenarios, etc. of the personal information involved will be informed to the user in an appropriate manner and the user's authorization will be obtained.

[0086] For example, when responding to an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested by the user will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, an application program, a server, or a storage medium that performs the operations of the present disclosure's technical solution based on the prompt message.

[0087] As an optional but non-limiting implementation manner, when responding to an active request from a user, the manner of sending a prompt message to the user can be, for example, in the form of a pop-up window. The prompt message can be presented in text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0088] It can be understood that the above process of notifying and obtaining user authorization is only illustrative and does not limit the implementation manner of the present disclosure. Other manners that comply with relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0089] Those of ordinary skill in the art should understand that: the discussion of any of the above embodiments is only exemplary and is not intended to imply that the scope of the present application is limited to these examples; under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations in different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of brevity.

[0090] In addition, for the sake of simplicity of description and discussion, and in order not to make the embodiments of the present application difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. In addition, the device can be shown in block diagram form to avoid making the embodiments of the present application difficult to understand, and this also takes into account the fact that the details of the implementation manner of these block diagram devices are highly dependent on the platform on which the embodiments of the present application will be implemented (that is, these details should be completely within the understanding of those skilled in the art). In the case where specific details (such as circuits) are set forth to describe the exemplary embodiments of the present application, it will be apparent to those skilled in the art that the embodiments of the present application can be implemented without these specific details or with variations of these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0091] Although the present application has been described in connection with specific embodiments of the present application, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other memory architectures (such as dynamic RAM (DRAM)) can be used with the embodiments discussed.

[0092] Embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of the present application shall be included within the protection scope of the present application.

Claims

1. A panoramic monitoring method for industrial equipment, characterized in that: include: Acquire in real time video data captured by multiple cameras corresponding to each device in a predetermined area, wherein the shooting angles of the multiple cameras corresponding to the device cover all angles of the device; For each device, multiple video data corresponding to the device are stitched together to obtain a panoramic video image; Based on the panoramic video image, 3D modeling is performed on the device to obtain a 3D device image; The 3D device image is displayed on the monitoring display interface.

2. The method according to claim 1, characterized in that The step of stitching the multiple video data corresponding to the device to obtain a panoramic video image includes: For each video data corresponding to the device, extract features of the video data to obtain a plurality of device features corresponding to the video data; For each device feature, searching for a device feature associated with the device feature among all device features corresponding to the device to form a device feature group; Based on all device feature groups, all device features are stitched using a predetermined image stitching technology to obtain the panoramic video image.

3. The method according to claim 1, characterized in that: The step of performing 3D modeling on the device based on the panoramic video image to obtain a 3D device image includes: Searching for a device three-dimensional model corresponding to the device in a pre-built device three-dimensional model library; The detail information in the panoramic video image is mapped to the device three-dimensional model to obtain the 3D device image.

4. The method according to claim 1, characterized in that: After the monitoring display interface displays the 3D device image, the method includes: In response to receiving an operation instruction for the 3D device image, the 3D device image is controlled to be displayed according to the operation instruction.

5. The method according to claim 1, characterized in that After the monitoring display interface displays the 3D device image, the method further includes: According to the time when the 3D device image is displayed on the monitoring display interface, the 3D device image, as well as the multiple video data and panoramic video images corresponding to the 3D device image are associated and archived into a pre-constructed database.

6. The method according to claim 5, characterized in that The method further comprises: In response to determining that the device fails, obtaining a failure time when the device fails; The database is searched for the 3D device image corresponding to the fault moment and each moment whose time difference with the fault moment is within a predetermined range, and multiple video data and panoramic video images corresponding to each 3D device image.

7. The method according to claim 1, characterized in that The method further comprises: In response to receiving a camera parameter adjustment instruction, parameters of the camera corresponding to the camera parameter adjustment instruction are adjusted.

8. A panoramic monitoring device for industrial equipment, characterized in that: include: An acquisition module is configured to acquire in real time video data captured by multiple cameras corresponding to each device in a predetermined area, wherein the shooting angles of the multiple cameras corresponding to the device cover all angles of the device; A splicing processing module is configured to perform splicing processing on multiple video data corresponding to each device to obtain a panoramic video image; A modeling module is configured to perform 3D modeling on the device based on the panoramic video image to obtain a 3D device image; The display module is configured to display the 3D device image on a monitoring display interface.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images