Station management method and system based on video fusion

Abstract

The invention discloses a station management method and system based on video fusion, and relates to the technical field of intelligent management, and the method comprises the steps: carrying out the three-dimensional modeling of a target station based on the three-dimensional measurement information of the target station, and generating a three-dimensional scene model; the panoramic video data are mapped to the three-dimensional scene model for fusion, and a three-dimensional scene fusion model is generated; performing three-dimensional inspection on the target station through the three-dimensional scene fusion model, and extracting an abnormal information set; fault positioning is carried out, and an alarm instruction is generated according to fault positioning information; and a remote control terminal is connected, and the alarm instruction is sent to the remote control terminal for intelligent management of the target station. The technical problems that in the prior art, a station lacks intelligent management, and operation safety and reliability are insufficient are solved, and the technical effects of intelligent and automatic management of the target station and improvement of safety and reliability are achieved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent management technology, and in particular to a site management method and system based on video fusion. Background Art

[0002] With the rapid development of industry and society, various stations play a vital role in production, transportation, energy supply and other fields. In the past, station management mainly relied on manual inspections and regular maintenance. Staff needed to check each area on site and record information such as equipment operation status and material storage conditions. For large stations, there are blind spots in monitoring, which means that some problems cannot be discovered in time.

[0003] The existing technology has technical problems such as lack of intelligent management of stations, insufficient operational safety and reliability. Summary of the invention

[0004] The present application provides a method and system for managing a station based on video fusion, which is used to solve the technical problems of lack of intelligent management, insufficient operational safety and reliability in the prior art.

[0005] In view of the above problems, the present application provides a site management method and system based on video fusion.

[0006] In a first aspect of the present application, a site management method based on video fusion is provided, the method comprising:

[0007] Based on the three-dimensional measurement information of the target station, the target station is three-dimensionally modeled to generate a three-dimensional scene model of the target station; multiple video acquisition points in the target station are traversed to determine multiple video sources; panoramic video data of the target station is constructed by real-time stitching based on the multiple video sources; the panoramic video data is mapped to the three-dimensional scene model for fusion to generate a three-dimensional scene fusion model; a three-dimensional inspection of the target station is performed through the three-dimensional scene fusion model, and an abnormal information set is extracted according to the inspection results; faults of the target station are located according to the abnormal information set, and an alarm instruction is generated according to the fault location information; a remote control terminal is connected, and the alarm instruction is sent to the remote control terminal for intelligent management of the target station.

[0008] The second aspect of the present application provides a site management system based on video fusion, the system comprising:

[0009] A three-dimensional scene model generation module, the three-dimensional scene model generation module performs three-dimensional modeling of the target station based on the three-dimensional measurement information of the target station to generate a three-dimensional scene model of the target station; a multiple video source determination module, the multiple video source determination module is used to traverse multiple video acquisition points in the target station to determine multiple video sources; a panoramic video data construction module, the panoramic video data construction module performs real-time splicing to construct panoramic video data of the target station based on the multiple video sources; a three-dimensional scene fusion model generation module, the three-dimensional scene fusion model generation module is used to use the panoramic video data to map the three-dimensional scene model for fusion, and generate a three-dimensional scene fusion model; a three-dimensional inspection of the target station is performed through the three-dimensional scene fusion model, and an abnormal information set is extracted according to the inspection results; an alarm instruction generation module, the alarm instruction generation module is used to locate the fault of the target station according to the abnormal information set, and generate an alarm instruction according to the fault location information; an intelligent management module, the intelligent management module is used to connect to a remote control terminal and send the alarm instruction to the remote control terminal for intelligent management of the target station.

[0010] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0011] Based on the three-dimensional measurement information of the target station, the target station is three-dimensionally modeled to generate a three-dimensional scene model of the target station, and the panoramic video data of the target station is constructed by real-time splicing based on the multiple video sources; the panoramic video data is mapped to the three-dimensional scene model for fusion to generate a three-dimensional scene fusion model; the target station is inspected in three dimensions through the three-dimensional scene fusion model, and an abnormal information set is extracted according to the inspection results; the target station is fault-located according to the abnormal information set, and an alarm instruction is generated according to the fault location information; the remote control terminal is connected, and the alarm instruction is sent to the remote control terminal for intelligent management of the target station. The intelligent and automated management of the target station is achieved, and the technical effect of improving safety and reliability is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0013] Figure 1 A schematic diagram of a flow chart of a site management method based on video fusion provided in an embodiment of the present application;

[0014] Figure 2A schematic diagram of a process for generating three-dimensional measurement information in a site management method based on video fusion provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of the structure of a site management system based on video fusion provided in an embodiment of the present application.

[0016] Explanation of the accompanying drawings: three-dimensional scene model generation module 10, multiple video source determination module 20, panoramic video data construction module 30, three-dimensional scene fusion model generation module 40, abnormal information set extraction module 50, alarm instruction generation module 60, intelligent management module 70. DETAILED DESCRIPTION

[0017] The present application provides a station management method and system based on video fusion, which is used to solve the technical problems of lack of intelligent management, insufficient operation safety and reliability of the prior art stations.

[0018] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] Embodiment 1

[0020] like Figure 1 As shown, the present application provides a site management method based on video fusion, the method comprising:

[0021] Step S100: performing three-dimensional modeling on the target station based on the three-dimensional measurement information of the target station to generate a three-dimensional scene model of the target station.

[0022] Specifically, it is necessary to first obtain comprehensive and accurate three-dimensional measurement information of the target station. There are various ways to obtain this measurement information, such as using laser scanning technology, drone mapping, or a combination of traditional measurement tools and methods. The three-dimensional measurement information obtained should cover all aspects of the target station, including but not limited to the topography of the station, the shape and size of the building, the layout of the road, the location and form of facilities and equipment, etc. With these detailed measurement data, professional three-dimensional modeling software and technology are used for modeling. During the modeling process, the basic framework of the station is constructed according to the measurement data, including the undulations of the terrain, the three-dimensional structure of the building, etc. For the modeling of the building, not only its external shape should be accurately presented, but also the internal spatial structure should be considered if necessary. At the same time, details such as materials and textures will be added to the model to enhance the realism and visualization of the model. The final generated three-dimensional scene model of the target station can show the overall form and layout of the station in a three-dimensional and intuitive way, providing an accurate basic framework for subsequent video fusion and management work.

[0023] Step S200: traverse multiple video acquisition points in the target station to determine multiple video sources.

[0024] Specifically, all pre-set video acquisition points in the target station are checked and confirmed one by one to clearly understand the location distribution of the video acquisition equipment that has been deployed in the target station. These acquisition points are distributed in key areas, such as entrances and exits, important equipment storage areas, cargo loading and unloading areas, and crowded areas. By traversing these points, the system can determine the video acquisition device corresponding to each point and obtain the video signals they generate, that is, determine multiple video sources. In the process of determining the video source, it is also necessary to consider the type, parameters and performance of the video acquisition device, such as resolution, frame rate, viewing angle range, etc., to ensure that the acquired video source can meet the needs of subsequent processing and analysis.

[0025] Step S300: Constructing panoramic video data of the target station by real-time stitching based on the multiple video sources.

[0026] Specifically, feature extraction is performed on the screen of each video source. These features include significant visual features such as corners, edges, and textures in the image. Then, based on the extracted features, the overlapping areas between adjacent video source screens are found through the image matching algorithm. After determining the overlapping areas, image fusion technology is used to perform smooth transition processing on these overlapping parts to avoid obvious breaks or distortions at the splicing points. At the same time, the time synchronization of the video source needs to be considered to ensure that there will be no picture jumps or delays during the splicing process. By continuously performing the above processing on multiple video sources, they are spliced ​​together in real time, and finally a panoramic video data that can fully display the global situation of the target station is formed, realizing all-round monitoring of the target station without blind spots, avoiding monitoring blind spots, improving the level of safety assurance, and accurately locating once an abnormality is found.

[0027] Step S400: Mapping the panoramic video data to the three-dimensional scene model for fusion to generate a three-dimensional scene fusion model.

[0028] Specifically, the panoramic video data constructed previously is combined with the previously generated three-dimensional scene model. First, a spatial correspondence between the panoramic video data and the three-dimensional scene model is established, which requires accurate matching and calibration of the pixel coordinates in the video data and the spatial coordinates in the three-dimensional scene model. Then, according to the established correspondence, the image information in the panoramic video data is accurately mapped to the corresponding surface or position of the three-dimensional scene model. During the fusion process, the illumination, color and other information of the video data are also considered, and the three-dimensional scene model is rendered and adjusted accordingly to make the fusion effect more realistic and natural. The final generated three-dimensional scene fusion model not only retains the precise geometric structure and spatial relationship of the three-dimensional scene model, but also has the real-time dynamics and rich details of the panoramic video data, enhances the visualization effect, provides a more immersive and realistic visualization experience, improves emergency response capabilities, and helps to make more accurate decisions.

[0029] Step S500: Perform a three-dimensional inspection on the target site using the three-dimensional scene fusion model, and extract an abnormal information set based on the inspection results.

[0030] Specifically, the generated 3D scene fusion model is used to conduct a comprehensive 3D inspection of the target station. Based on the preset inspection rules and standards, the human perspective and action path will be simulated in the 3D scene fusion model to carefully inspect each part of the station. During the inspection process, various elements in the model will be analyzed and compared in real time, including the operating status of the equipment, the placement of items, the activities of personnel, etc. According to the analysis and judgment results during the inspection process, all information that does not conform to the normal state is extracted, and these abnormal information is sorted and summarized into an abnormal information set. The speed and efficiency of the inspection are greatly improved, subtle abnormalities are discovered more accurately, the probability of missed detection and false detection is reduced, and real-time monitoring of the station status and timely extraction of abnormal information are realized, which improves the safety and reliability of station operation.

[0031] Step S600: locating the fault of the target station according to the abnormal information set, and generating an alarm instruction according to the fault locating information.

[0032] Specifically, the previously extracted abnormal information set is analyzed in depth. According to the various data and features contained in the abnormal information set, such as the location, type, and degree of the abnormality, the specific location of the fault in the target station is accurately determined by using relevant positioning algorithms and technologies (such as Bluetooth positioning technology, map matching technology, inertial navigation technology, etc.). After the fault location is determined, the nature of the fault, the possible impact, and the preset alarm rules are comprehensively considered to generate the corresponding alarm instructions. The alarm instruction usually contains key contents such as a detailed description of the fault, location information, and urgency, so that relevant personnel can quickly and accurately understand the fault situation. For example, if the abnormal information set indicates that the temperature in a certain area is too high, after analysis, it is determined that it is caused by a failure in the cooling system of a specific device, and an alarm instruction for the location and fault type of the device will be generated, which can quickly locate the fault and generate an alarm instruction, so that relevant personnel can know the fault situation at the first time, so as to respond quickly and reduce the losses caused by the fault. Timely alarms can prevent small faults from turning into big problems, reduce the severity and scope of the accident, and improve safety and reliability.

[0033] Step S700: connecting to a remote control terminal, and sending the alarm instruction to the remote control terminal for intelligent management of the target site.

[0034] Specifically, an effective connection is established with the remote control terminal through a network communication protocol, such as a wireless network, a wired network, or a dedicated communication link. Once the connection is successfully established, the previously generated alarm command will be quickly and accurately sent to the remote control terminal, which can be a mobile device, such as a mobile phone, a tablet computer, or a computer in a fixed control center. When the remote control terminal receives the alarm command, relevant personnel, such as station managers, maintenance personnel, or monitoring personnel, can obtain detailed information about the target station failure in a timely manner, and make corresponding operations and management decisions remotely based on this information, such as remotely starting emergency plans, dispatching maintenance personnel, and adjusting equipment operating parameters. This step makes the management of the target station no longer limited by geographical location, improves the timeliness of management and the safety of the station, and realizes the intelligent and automated management of the target station.

[0035] In one possible implementation, Figure 2 As shown, the step S100 also includes:

[0036] Step S110: Perform a comprehensive scan of the target station based on remote sensing technology to generate three-dimensional spatial structure information. Step S120: Use drones to traverse the target station and generate three-dimensional spatial position information. Step S130: Generate three-dimensional spatial terrain information by performing field measurements on the target station. Step S140: Add the three-dimensional spatial structure information, the three-dimensional spatial position information, and the three-dimensional spatial terrain information to the three-dimensional measurement information.

[0037] Specifically, remote sensing technology is used to conduct a comprehensive scan of the target station. Remote sensing technology usually uses sensors carried by satellites, aircraft, etc. to collect electromagnetic wave data of the target station. These sensors can capture electromagnetic waves of different wavelengths and frequencies, including visible light, infrared, microwaves, etc. By processing and analyzing the large amount of electromagnetic wave data collected, the preliminary three-dimensional spatial structure information of the target station is constructed. This structural information includes the overall outline of the station, the approximate shape and layout of the main buildings and facilities, the division of large sites, etc. The advantage of this step is that it can quickly and widely obtain the basic information of the target station, and is not limited by ground conditions and the complex environment inside the station.

[0038] Use drones to traverse the target station and generate three-dimensional spatial location information, mainly including the location of each device in the station. According to the scope and characteristics of the target station, plan the flight route and shooting points of the drone to ensure that all areas of the station can be fully covered. Then control the drone equipped with a high-resolution camera or professional photogrammetry equipment to take aerial photos according to the planned route. During the flight, a large number of images containing equipment in the station are obtained by shooting at different angles and heights. The acquired images are preprocessed, including noise removal, distortion correction and other operations to improve image quality. After that, professional image processing software and algorithms are used to analyze and process the preprocessed images. Through feature extraction, image matching and other technologies, the coordinate position of each device in three-dimensional space is accurately calculated. Finally, the generated three-dimensional spatial location information is integrated with other measurement information (such as three-dimensional spatial structure information, three-dimensional spatial terrain information, etc.) to form complete three-dimensional measurement information.

[0039] Use specific measuring instruments and tools to go to the target station for field measurement. The surveyors plan the measurement route and point layout according to the terrain characteristics and measurement requirements of the target station. Common measuring instruments include total stations, levels, GPS receivers, etc. When using total stations, surveyors aim at the target point and measure parameters such as horizontal angles, vertical angles and distances to determine the position of each point. Levels are mainly used to measure the elevation of each point and obtain the ups and downs of the terrain. For GPS receivers, it can directly receive satellite signals and quickly obtain the precise position and elevation information of the measurement point. During the measurement process, surveyors will measure point by point according to the predetermined route and point layout plan, and record the coordinates (including plane coordinates and elevation), terrain features and other information of each measurement point. After the measurement is completed, the collected data will be imported into professional geographic information system (GIS) software or other data processing software. By processing and analyzing these data, a three-dimensional spatial terrain model of the target station is constructed. This model can accurately reflect the terrain characteristics of the station ground, such as ups and downs, slope, and slope direction.

[0040] In this step, the three-dimensional spatial structure information, three-dimensional spatial position information and three-dimensional spatial terrain information obtained separately are integrated together and added to the overall three-dimensional measurement information. First, a unified data format and coordinate system need to be established to ensure that information from different sources can be accurately matched and fused. For the three-dimensional spatial structure information, the data such as the overall layout of the station, the approximate shape of buildings and facilities contained therein are converted and sorted according to a unified format and coordinate standard. For the three-dimensional spatial position information, the precise coordinate position data of each device in the station is matched with the overall coordinate system, and calibration and correction are performed to ensure the accuracy of the position. The three-dimensional spatial terrain information is also processed according to a unified standard so that it can perfectly fit with other information in space. Then, through data fusion algorithms and software tools, these processed information are added to the existing three-dimensional measurement information in turn. During the addition process, data verification and inspection should be carried out to ensure that there are no conflicts and errors in the newly added information. Finally, a complete, rich and accurate three-dimensional measurement information set is formed, which contains the comprehensive spatial characteristics of the target station. It provides a more accurate and comprehensive basis for the management and planning of the station, which helps to make more scientific decisions.

[0041] In a possible implementation, step S200 further includes:

[0042] Step S210: Extract multiple video streams based on the multiple video sources for time synchronization to generate multiple synchronized video streams. Step S220: Perform perspective transformation based on the multiple video acquisition points of the multiple synchronized video streams, extract features from the multiple synchronized video streams in combination with the transformation results, and determine multiple feature areas. Step S230: Match and align the multiple synchronized video streams according to the multiple feature areas to determine multiple stitching information, wherein the multiple stitching information includes multiple stitching positions and multiple stitching angles. Step S240: Perform real-time stitching and synthesis of the multiple synchronized video streams according to the multiple stitching positions and in combination with the multiple stitching angles to generate the panoramic video data.

[0043] Specifically, first obtain the corresponding video stream data from multiple video sources. Since these video sources may come from different devices and their acquisition and transmission processes are different, there may be time deviations between the video streams. In order to achieve time synchronization, it is necessary to analyze the time stamps or related time information in each video stream. This time information may be embedded in the metadata of the video or transmitted through a specific protocol. Use a special time synchronization algorithm to adjust and calibrate these video streams. For example, take one of the video streams as a benchmark, calculate the time difference between other video streams and the benchmark video stream, and delay or advance other video streams based on this difference so that they can be aligned on the timeline. In the process of time synchronization, factors such as the frame rate and acquisition frequency of the video stream also need to be considered to ensure the accuracy and stability of synchronization.

[0044] The specific positions and shooting angles of the multiple video acquisition points corresponding to the multiple synchronous video streams are clarified. Since different acquisition points have different viewing angles and distances, the objects in the video are deformed in perspective. In order to eliminate this deformation difference, multiple synchronous video streams need to be transformed in perspective. Perspective transformation is a geometric transformation. According to the shooting angle and position of each video source, the video stream is transformed in perspective, that is, pre-processing operations such as distortion correction and color correction are performed to eliminate the deformation caused by different shooting angles. After completing the perspective transformation, the transformed multiple synchronous video streams are then subjected to feature extraction. This can be achieved by using various image processing algorithms, such as edge detection algorithms to extract the edge contours of objects, corner detection algorithms to determine significant corners in the image, or texture analysis-based methods to extract areas with unique textures. By extracting features from multiple synchronous video streams, multiple feature areas can be determined. These feature areas may be the edges of objects, parts with specific textures, and areas with significant corner concentrations.

[0045] Based on the multiple feature regions extracted above, the matching and alignment operation of multiple synchronous video streams is performed. The feature regions in different video streams are compared and analyzed one by one, and the corresponding relationship between them is found through factors such as feature similarity and correlation. For example, if an edge feature in one video stream has a high degree of similarity with the corresponding edge feature in another video stream in terms of shape, length, direction, etc., the two features can be considered to be matched. After determining the matching relationship of the feature regions, the relative position and angle relationship between the multiple synchronous video streams are further calculated based on these matching results, and multiple splicing positions and multiple splicing angles are accurately determined. The splicing position refers to the specific coordinate position of the overlapping part of the image when different video streams are spliced. The splicing angle refers to the angle that one video stream needs to be rotated relative to another video stream when splicing to achieve a smooth and seamless splicing effect. For example, if you want to splice the video streams shot by two adjacent cameras, through feature matching, it is found that their overlapping part is mainly a corner of a building, and then the specific position of this overlapping part in the two video streams and the angle that needs to be rotated are calculated to determine the splicing position and splicing angle.

[0046] Based on the multiple stitching positions and multiple stitching angles determined previously, multiple synchronous video streams are stitched together in real time. According to the information of the stitching position, the corresponding image parts in the multiple video streams are accurately overlapped, which requires precise image positioning and alignment technology to ensure that each pixel can be combined according to the predetermined position. At the same time, combined with the information of the stitching angle, the corresponding video streams are rotated and adjusted so that they can also perfectly match in angle. In the process of stitching and synthesis, the images at the stitching are also fused, including the use of image smoothing algorithms, color correction and other technologies to eliminate obvious traces of the stitching edges, so that the transition between different video streams is more natural and smooth. By performing these operations in real time, multiple synchronous video streams are seamlessly stitched into a whole, and finally panoramic video data is generated. The scattered video streams are integrated into a unified panoramic video data for subsequent storage, analysis and utilization.

[0047] In a possible implementation, step S400 further includes:

[0048] Step S410: Feature identification is performed on the three-dimensional scene model according to the multiple feature areas to determine multiple feature identification points. Step S420: The association relationship between the multiple feature identification points is calculated to generate multiple feature association information. Step S430: A mapping variable relationship between the panoramic video data and the three-dimensional scene model is established based on the multiple feature association information. Step S440: Extract the target video frame sequence of the panoramic video data, and project the target video frame sequence to the three-dimensional scene model according to the mapping variable relationship for fusion to generate the three-dimensional scene fusion model.

[0049] Specifically, according to the multiple feature regions obtained previously, corresponding feature identification is performed on the 3D scene model. These feature regions may be parts with significant features extracted from the video stream, such as specific objects, textures or geometric shapes. By matching and marking these features with corresponding parts in the 3D scene model, multiple feature identification points are determined.

[0050] The association relationship between these feature identification points is determined through a series of calculation and analysis methods. This may include but is not limited to the following methods: spatial distance calculation, directional relationship analysis, topological relationship judgment, etc. Through the comprehensive analysis and calculation of the above various methods, multiple feature association information is generated, which describes in detail the relationship between the feature identification points, such as which points are closely related, which points have indirect connections, and which points are based on specific conditions.

[0051] Based on the obtained multiple feature association information, a mapping variable relationship between the panoramic video data and the three-dimensional scene model is created. These feature association information is deeply analyzed and understood. Determine which feature associations correspond to specific elements in the panoramic video data, as well as their corresponding positions and attributes in the three-dimensional scene model. Then, these corresponding relationships are converted into specific mapping rules, which define how to accurately map each pixel and each object in the panoramic video data to the corresponding position of the three-dimensional scene model based on the feature association information. The mapping variable relationship includes rules for spatial coordinate conversion, color and brightness adjustment, texture matching, etc. For example, the position and posture of an object in the panoramic video are determined based on the feature association, and then the corresponding coordinate transformation in the three-dimensional scene model is calculated, as well as how to apply the color and texture information in the video to the corresponding surface of the model. In the process of establishing the mapping variable relationship, it is also necessary to take into account possible errors and uncertainties, and perform appropriate optimization and correction to ensure the accuracy and stability of the mapping.

[0052] Carefully select target video frame sequences with key information and representativeness from the panoramic video data, such as frames containing important objects, significant events or specific time periods. Then, based on the previously established mapping variable relationship, each pixel and element in the target video frame sequence is accurately converted and projected, and it is necessary to ensure that they can perfectly match in terms of shape, position, color, etc. During the fusion process, the dynamic changes of the video frames and the static structure of the 3D scene model are taken into account, and complex calculations and processing are performed to achieve an organic combination of the two. Through such a fusion operation, a new 3D scene fusion model is finally generated. Abnormal situations can be judged more quickly and accurately, and corresponding response strategies can be formulated.

[0053] In a possible implementation, step S500 further includes:

[0054] Step S510: Determine multiple inspection information according to the inspection target parameters, and the multiple inspection information includes the inspection starting point position information, the inspection end point position information, the target inspection perspective, and the target inspection object information. Step S520: Randomly generate an inspection path according to the inspection starting point position information and the inspection end point position information, add the target inspection perspective and the target inspection object information to the inspection path, and determine the target inspection path. Step S530: Perform automatic inspection according to the target inspection path based on the three-dimensional scene fusion model to obtain an inspection result, and the inspection result includes the three-dimensional scene inspection position information and the three-dimensional scene inspection perspective information. Step S540: Perform abnormal analysis according to the three-dimensional scene inspection position information and the three-dimensional scene inspection perspective information, identify according to the abnormal analysis results, and determine the abnormal identification position information, abnormal identification perspective information, and abnormal identification object information. Step S550: Add the abnormal identification position information, the abnormal identification perspective information, and the abnormal identification object information to the abnormal information set.

[0055] Specifically, the inspection target parameters are first set according to the specific inspection needs. If it is desired to understand the overall operation of the equipment in the station, the inspection target parameters will be set to cover all areas and equipment in the entire station. Based on these inspection target parameters, multiple detailed inspection information is determined. The inspection starting point location information will be set to the location where a comprehensive inspection can be started most effectively, which may be the main entrance of the station, the entrance to the key equipment concentration area, etc. The inspection end point location information is correspondingly set to the location that can complete the coverage of all key areas and equipment in the entire station to ensure that nothing is missed. The target inspection perspective will be determined based on the need to comprehensively observe the operation of the equipment, which may include observations from different heights and angles to obtain operating status information of various aspects of the equipment. The target inspection object information will be clearly defined as all important equipment within the site to ensure that every device that may affect the overall operation is inspected. If you only want to understand the situation in a certain area, such as certain types of equipment in a specific area, then the inspection target parameters will focus on these specific areas and equipment. At this time, the inspection starting point location information is the entrance or nearby key location of the specific area, and the inspection end point location information is the location after completing the inspection of the key equipment in this area. The target inspection perspective will also be set according to the characteristics of these specific equipment and the location of common problems. The target inspection object information will be clearly defined as the specified types of equipment in this area.

[0056] A random algorithm is used to generate the inspection path based on the determined inspection starting point location information and inspection end point location information. The randomness here is not completely disordered, but is carried out under certain constraints, such as avoiding obstacles and following specific channel rules. After the initial random path is generated, the target inspection perspective and target inspection object information are added to the path. This means that in path planning, not only the starting point and the end point should be considered, but also the predetermined target inspection perspective should be achieved at the location where the path passes, and the target inspection object can be covered. By comprehensively considering these factors, the initially generated path is adjusted and optimized, and the target inspection path is finally determined to ensure a comprehensive and accurate inspection of the target object and improve the quality of the inspection.

[0057] In this step, based on the constructed 3D scene fusion model, the automatic inspection operation is carried out according to the previously determined target inspection path. During the inspection process, relevant information will be recorded in real time to obtain the inspection results. Among them, the 3D scene inspection position information records the specific position coordinates passed on the inspection path, accurately reflecting the position in the 3D space. The 3D scene inspection perspective information records the observation angle and direction taken at each inspection position.

[0058] First, obtain the three-dimensional scene inspection position information and the three-dimensional scene inspection perspective information. Then, based on these detailed information, perform an abnormal analysis on the data obtained during the inspection. Abnormal analysis involves many methods, such as comparing the currently acquired data with the preset normal range or standard model, or using image recognition, data analysis and other technologies to detect whether there are any unexpected situations. Clearly identify according to the results of the abnormal analysis. Determine the abnormal identification position information, that is, the specific three-dimensional spatial position coordinates where the abnormal situation occurs; abnormal identification perspective information, that is, the perspective direction where the abnormal situation can be observed most clearly; abnormal identification object information, that is, the specific object or element where the abnormality occurs.

[0059] The above-determined abnormal identification position information, abnormal identification viewing angle information and abnormal identification object information are added to the abnormal information set. The purpose of this is to centrally manage and store all detected abnormality-related information to facilitate subsequent further processing, analysis and decision-making.

[0060] In a possible implementation, step S600 further includes:

[0061] Step S610: Extract fault type information and fault level information according to the alarm instruction. Step S620: Weight the fault location information in combination with the fault type information and the fault level information to generate a fault weight distribution result. Step S630: Use the fault weight distribution result as an index to traverse the abnormal response strategy library to determine and formulate an abnormal response plan. Step S640: Transmit the abnormal response plan to the remote control terminal.

[0062] Specifically, first, when an alarm instruction is received, fault type information (such as equipment damage, system error, network interruption, etc.) and fault level information (such as minor, serious, urgent, etc.) are extracted therefrom.

[0063] Then, the fault location information is weighted based on the acquired fault type and fault level information. For example, a serious network outage fault may be assigned a higher weight, while a minor device display abnormality may be assigned a lower weight. The generated fault weight distribution results can reflect the relative importance and urgency of different faults.

[0064] Next, the fault weight distribution result is used as an index to search in the abnormal response strategy library. The abnormal response strategy library pre-stores response plans for various possible situations. According to the weight distribution result, a matching strategy is found to determine and formulate a specific abnormal response plan.

[0065] Finally, the determined abnormal response plan is transmitted to the remote control terminal so that relevant personnel can obtain it in time and take corresponding processing measures for efficient fault handling.

[0066] In a possible implementation, step S700 further includes:

[0067] Step S710: Combined with the alarm instruction, the remote control terminal is used to review the fault type information and the fault level information to generate alarm review data. Step S720: Based on the alarm review data, it is determined whether the alarm review fault level is greater than or equal to the preset fault level. Step S730: If the alarm review fault level is greater than or equal to the preset fault level, a reverse query instruction is generated, and the three-dimensional scene fusion model is backtracked according to the reverse query instruction, and the backtracking result is fed back to the remote control terminal.

[0068] Specifically, after receiving the alarm instruction, the remote control terminal will obtain the fault type information and fault level information contained therein. Then, the relevant personnel will use the functions and tools of the remote control terminal to further verify and review this information. This may include the following operations: View the detailed monitoring data, images or video materials related to the fault to confirm the accuracy of the fault type. For example, if the alarm instruction shows that it is a short circuit fault in the power system, then the current, voltage and other data will be checked to confirm whether it is indeed a short circuit. Evaluate the actual impact scope and severity caused by the fault and compare it with the fault level information initially given. For example, for a network fault that is claimed to be a serious level, the number of affected users and the interruption of key services will be checked to determine whether the level is appropriate. Collect more relevant information on the site or in the surrounding area, which may be supplemented and improved by communicating with on-site personnel and obtaining data from other sensors. Combined with the above verification and review work, detailed and accurate alarm review data is generated. This data not only includes reconfirmation of the fault type and level, but also may include newly discovered relevant details, corrections or supplements to the initial information, etc.

[0069] The step of judging whether the alarm review fault level is greater than or equal to the preset fault level based on the alarm review data is intended to determine whether the severity of the current fault reaches or exceeds the preset standard. By comparing the fault level in the alarm review data with the preset fault level, the urgency and importance of the fault can be quickly determined. If the alarm review fault level is greater than or equal to the preset fault level, it may mean that more urgent and serious measures need to be taken to solve the problem, such as starting a reverse query instruction to backtrack to further determine the cause of the fault and the scope of impact.

[0070] If it is determined that the alarm review fault level is greater than or equal to the preset fault level, this indicates that the current fault situation is more serious and requires more in-depth investigation and analysis. At this point, the system will automatically generate a reverse query instruction. This reverse query instruction will serve as a trigger signal for the operation and is used to perform a backtracking operation on the three-dimensional scene fusion model. During the backtracking process, the system will work backwards along the timeline to find the state and related data of the three-dimensional scene fusion model for a period of time before the fault occurred. This may include information such as changes in the operating parameters of the equipment, the position and state changes of objects in the scene, etc. After the backtracking is completed, the backtracking results will be fed back to the remote control terminal. These results will be presented in the form of data, images, charts, etc., providing relevant personnel with detailed information and possible clues before the fault occurred.

[0071] Embodiment 2

[0072] Based on the same inventive concept as the site management method based on video fusion in the aforementioned embodiment, Figure 3 As shown, the present application provides a site management system based on video fusion, and the system and method embodiments in the present application embodiments are based on the same inventive concept. The system includes:

[0073] The three-dimensional scene model generation module 10 performs three-dimensional modeling on the target station based on the three-dimensional measurement information of the target station to generate a three-dimensional scene model of the target station.

[0074] A plurality of video source determination modules 20 are used to traverse a plurality of video acquisition points in a target station to determine a plurality of video sources.

[0075] The panoramic video data construction module 30 constructs the panoramic video data of the target station by real-time splicing based on the multiple video sources.

[0076] The three-dimensional scene fusion model generation module 40 is used to map the panoramic video data to the three-dimensional scene model for fusion, so as to generate a three-dimensional scene fusion model.

[0077] The abnormal information set extraction module 50 is used to perform a three-dimensional inspection on the target site through the three-dimensional scene fusion model, and extract the abnormal information set according to the inspection result.

[0078] The alarm instruction generating module 60 is used to locate the fault of the target station according to the abnormal information set and generate an alarm instruction according to the fault location information.

[0079] The intelligent management module 70 is used to connect to a remote control terminal and send the alarm instruction to the remote control terminal to perform intelligent management of the target station.

[0080] Furthermore, the 3D scene model generation module 10 also includes:

[0081] A three-dimensional spatial structure information generating unit, the three-dimensional spatial structure information generating unit performs a comprehensive scan of the target station based on remote sensing technology to generate three-dimensional spatial structure information. A three-dimensional spatial position information generating unit, the three-dimensional spatial position information generating unit uses drone aerial photography to traverse the target station to generate three-dimensional spatial position information. A three-dimensional spatial terrain information generating unit, the three-dimensional spatial terrain information generating unit is used to generate three-dimensional spatial terrain information by performing field measurements on the target station. A three-dimensional measurement information unit, the three-dimensional measurement information unit is used to add the three-dimensional spatial structure information, the three-dimensional spatial position information, and the three-dimensional spatial terrain information to the three-dimensional measurement information.

[0082] Furthermore, the panoramic video data construction module 30 also includes:

[0083] A plurality of synchronous video stream generating units, the plurality of synchronous video stream generating units extracting a plurality of video streams based on the plurality of video sources for time synchronization to generate a plurality of synchronous video streams. A plurality of feature region determining units, the plurality of feature region determining units performing perspective transformation based on the plurality of video acquisition points of the plurality of synchronous video streams, extracting features from the plurality of synchronous video streams in combination with the transformation results, and determining a plurality of feature regions. A matching alignment unit, the matching alignment unit being used to match and align the plurality of synchronous video streams according to the plurality of feature regions, and determining a plurality of stitching information, the plurality of stitching information comprising a plurality of stitching positions and a plurality of stitching angles. A panoramic video data generating unit, the panoramic video data generating unit being used to perform real-time stitching and synthesis of the plurality of synchronous video streams in combination with the plurality of stitching angles according to the plurality of stitching positions, and generating the panoramic video data.

[0084] Furthermore, the 3D scene fusion model generation module 40 also includes:

[0085] A plurality of feature identification point determination units, the plurality of feature identification point determination units perform feature identification on the three-dimensional scene model according to the plurality of feature areas to determine a plurality of feature identification points. A plurality of feature association information generation units, the plurality of feature association information generation units are used to calculate the association relationship between the plurality of feature identification points to generate a plurality of feature association information. A mapping variable relationship between the panoramic video data and the three-dimensional scene model is established based on the plurality of feature association information. A target video frame sequence of the panoramic video data is extracted, and the target video frame sequence is projected onto the three-dimensional scene model according to the mapping variable relationship for fusion to generate the three-dimensional scene fusion model.

[0086] Furthermore, the abnormal information set extraction module 50 also includes:

[0087] A plurality of inspection information determination units, wherein the plurality of inspection information determination units determine a plurality of inspection information according to inspection target parameters, wherein the plurality of inspection information includes inspection starting point position information, inspection end point position information, target inspection angle of view, and target inspection object information. A target inspection path determination unit, wherein the target inspection path determination unit randomly generates an inspection path according to the inspection starting point position information and the inspection end point position information, adds the target inspection angle of view and the target inspection object information to the inspection path, and determines the target inspection path. An inspection result acquisition unit, wherein the inspection result acquisition unit automatically inspects the target inspection path based on the three-dimensional scene fusion model, and obtains an inspection result, wherein the inspection result includes three-dimensional scene inspection position information and three-dimensional scene inspection angle of view information. An abnormal analysis result identification unit, wherein the abnormal analysis result identification unit is used to perform abnormal analysis according to the three-dimensional scene inspection position information and the three-dimensional scene inspection angle of view information, identify according to the abnormal analysis result, and determine abnormal identification position information, abnormal identification angle of view information, and abnormal identification object information. An exception information set adding unit, the exception information set adding unit is used to add the exception identification position information, the exception identification viewing angle information, and the exception identification object information to the exception information set.

[0088] Furthermore, the alarm instruction generating module 60 further includes:

[0089] An alarm instruction extraction unit, the alarm instruction extraction unit extracts fault type information and fault level information according to the alarm instruction. A fault weight allocation result generation unit, the fault weight allocation result generation unit weights the fault location information in combination with the fault type information and the fault level information to generate a fault weight allocation result. An abnormal response plan formulation unit, the abnormal response plan formulation unit is used to use the fault weight allocation result as an index, traverse the abnormal response strategy library, and determine to formulate an abnormal response plan. An abnormal response plan transmission unit, the abnormal response plan transmission unit is used to transmit the abnormal response plan to a remote control terminal.

[0090] Furthermore, the intelligent management module 70 also includes:

[0091] An alarm review data generation unit, the alarm review data generation unit combines the alarm instruction, and reviews the fault type information and the fault level information through the remote control terminal to generate alarm review data. An alarm review fault level judgment unit, the alarm review fault level judgment unit judges whether the alarm review fault level is greater than or equal to the preset fault level based on the alarm review data. A reverse check instruction generation unit, the reverse check instruction generation unit is used to generate a reverse check instruction if the alarm review fault level is greater than or equal to the preset fault level, and backtrack the three-dimensional scene fusion model according to the reverse check instruction, and feed back the backtracking result to the remote control terminal.

[0092] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0093] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0094] This specification and the drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. A site management method based on video fusion, characterized in that: The method comprises: Perform three-dimensional modeling of the target station based on the three-dimensional measurement information of the target station to generate a three-dimensional scene model of the target station; Traverse multiple video collection points in the target station to determine multiple video sources; Constructing panoramic video data of the target station by real-time stitching based on the multiple video sources; Mapping the panoramic video data to the three-dimensional scene model for fusion to generate a three-dimensional scene fusion model; Performing a three-dimensional inspection on the target station through the three-dimensional scene fusion model, and extracting an abnormal information set according to the inspection results; Perform fault location on the target station according to the abnormal information set, and generate an alarm instruction according to the fault location information; Connect to a remote control terminal and send the alarm instruction to the remote control terminal to perform intelligent management of the target site.

2. A site management method based on video fusion as claimed in claim 1, characterized in that: The three-dimensional measurement information method includes: Comprehensively scan the target station based on remote sensing technology to generate three-dimensional spatial structure information; Use drones to traverse target stations and generate three-dimensional spatial location information; Generate three-dimensional spatial terrain information by conducting field measurements on target stations; The three-dimensional space structure information, the three-dimensional space position information, and the three-dimensional space terrain information are added to the three-dimensional measurement information.

3. A site management method based on video fusion as claimed in claim 1, characterized in that: Based on the multiple video sources, real-time stitching is performed to construct panoramic video data of the target station, and the method includes: Extract multiple video streams based on the multiple video sources for time synchronization, and generate multiple synchronized video streams; Performing perspective transformation based on the multiple video acquisition points of the multiple synchronous video streams, extracting features from the multiple synchronous video streams based on the transformation results, and determining multiple feature areas; Matching and aligning the multiple synchronous video streams according to the multiple feature areas to determine multiple splicing information, where the multiple splicing information includes multiple splicing positions and multiple splicing angles; The multiple synchronous video streams are stitched and synthesized in real time according to the multiple stitching positions and in combination with the multiple stitching angles to generate the panoramic video data.

4. A site management method based on video fusion as claimed in claim 3, characterized in that: The panoramic video data is mapped to the three-dimensional scene model for fusion to generate a three-dimensional scene fusion model, the method comprising: Performing feature identification on the three-dimensional scene model according to the multiple feature areas to determine multiple feature identification points; Calculating the association relationship between the plurality of feature identification points to generate a plurality of feature association information; Establishing a mapping variable relationship between the panoramic video data and the three-dimensional scene model according to the plurality of feature association information; A target video frame sequence of the panoramic video data is extracted, and the target video frame sequence is projected onto the three-dimensional scene model according to the mapping variable relationship for fusion, so as to generate the three-dimensional scene fusion model.

5. The method for station management based on video fusion according to claim 1, characterized in that: The target station is inspected in three dimensions by using the three-dimensional scene fusion model, and an abnormal information set is extracted according to the inspection result. The method includes: Determine multiple inspection information according to the inspection target parameters, wherein the multiple inspection information includes inspection starting point location information, inspection end point location information, target inspection viewing angle, and target inspection object information; Randomly generate an inspection path according to the inspection starting point location information and the inspection end point location information, add the target inspection perspective and the target inspection object information to the inspection path, and determine the target inspection path; Based on the three-dimensional scene fusion model, automatic inspection is performed according to the target inspection path to obtain an inspection result, wherein the inspection result includes three-dimensional scene inspection position information and three-dimensional scene inspection viewing angle information; Perform an abnormality analysis according to the three-dimensional scene inspection position information and the three-dimensional scene inspection viewing angle information, identify the abnormality according to the abnormality analysis result, and determine the abnormal identification position information, abnormal identification viewing angle information, and abnormal identification object information; The abnormal identification position information, the abnormal identification viewing angle information, and the abnormal identification object information are added to the abnormal information set.

6. A site management method based on video fusion as claimed in claim 1, characterized in that: Methods include: Extracting fault type information and fault level information according to the alarm instruction; weighting the fault location information in combination with the fault type information and the fault level information to generate a fault weighting result; Using the fault weight distribution result as an index, traversing the abnormal response strategy library, and determining to formulate an abnormal response plan; The abnormal response scheme is transmitted to the remote control terminal.

7. A site management method based on video fusion as claimed in claim 6, characterized in that: The alarm instruction is sent to the remote control terminal to perform intelligent management of the target station, the method comprising: In combination with the alarm instruction, the remote control terminal conducts a review according to the fault type information and the fault level information to generate alarm review data; Based on the alarm review data, determining whether the alarm review fault level is greater than or equal to a preset fault level; If the alarm review fault level is greater than or equal to the preset fault level, a reverse query instruction is generated, the three-dimensional scene fusion model is backtracked according to the reverse query instruction, and the backtracking result is fed back to the remote control terminal.

8. A station management system based on video fusion, characterized in that: The system is used to implement a site management method based on video fusion according to any one of claims 1 to 7, and the system comprises: A three-dimensional scene model generation module, wherein the three-dimensional scene model generation module performs three-dimensional modeling on the target station based on the three-dimensional measurement information of the target station to generate a three-dimensional scene model of the target station; A plurality of video source determination modules, wherein the plurality of video source determination modules are used to traverse a plurality of video acquisition points in a target station to determine a plurality of video sources; A panoramic video data construction module, wherein the panoramic video data construction module constructs the panoramic video data of the target station by real-time splicing based on the multiple video sources; A three-dimensional scene fusion model generation module, wherein the three-dimensional scene fusion model generation module is used to map the panoramic video data to the three-dimensional scene model for fusion to generate a three-dimensional scene fusion model; An abnormal information set extraction module, the abnormal information set extraction module is used to perform a three-dimensional inspection of the target station through the three-dimensional scene fusion model, and extract the abnormal information set according to the inspection result; An alarm instruction generation module, the alarm instruction generation module is used to locate the fault of the target station according to the abnormal information set, and generate an alarm instruction according to the fault location information; An intelligent management module, wherein the intelligent management module is used to connect to a remote control terminal and send the alarm instruction to the remote control terminal to perform intelligent management of the target site.

Images