Anomaly positioning method and system based on panoramic video fusion

Abstract

The present application relates to the field of abnormality positioning technology, and provides an abnormality positioning method and system based on panoramic video fusion. The method includes: collecting and returning discrete video streams of the control area to determine multiple video streams; converting and clustering the multiple video streams to determine the pre-processed video streams; constructing a three-dimensional panoramic fusion module, fusion and splicing modeling of the pre-processed video streams, and determining the fused panorama; based on the fused panorama, locating abnormal events and refining the viewing angle to determine abnormal positioning data; determining the spatial distribution field, performing spatial field solution for the abnormal positioning data, and determining the target camera pan-tilt; controlling the target camera pan-tilt, performing refined monitoring, and determining the abnormal positioning column. The present application solves the technical problems of limited monitoring range and low positioning accuracy in traditional abnormality positioning methods, and realizes full coverage, high-precision monitoring and positioning of the control area through panoramic video fusion, thereby improving the efficiency of abnormal event reverse investigation.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, specifically to the field of abnormality positioning technology, and more particularly to an abnormality positioning method and system based on panoramic video fusion. Background Art

[0002] With the rapid development of industrial production and increasing automation, the timely detection and location of equipment failures and major emergency events in industrial environments has become crucial. These events can not only disrupt production but also have serious impacts on personnel safety and the environment. However, traditional methods for locating industrial anomalies rely on a combination of regular manual inspections and monitoring equipment. However, this model suffers from low positioning accuracy and slow response speeds due to human factors and blind spots in monitoring, making it difficult to meet the efficiency, precision, and safety requirements of modern industrial production. Summary of the Invention

[0003] This application provides an anomaly positioning method and system based on panoramic video fusion, aiming to solve the technical problems of limited monitoring range and low positioning accuracy in traditional anomaly positioning methods.

[0004] In view of the above problems, the present application provides an anomaly positioning method and system based on panoramic video fusion.

[0005] The first aspect disclosed in the present application provides an anomaly location method based on panoramic video fusion, the method comprising: based on a distributed camera gimbal, collecting and returning discrete video streams of a controlled area to determine multiple video streams, wherein the multiple video streams are timestamp-identified and synchronized based on a network time protocol; performing coaxial conversion and homologous clustering on the multiple video streams to determine a preprocessed video stream, wherein each cluster corresponds to a local video stream and is identified with a relative spatial position; constructing a three-dimensional panoramic fusion module, performing fusion and splicing modeling on the preprocessed video streams to determine a fused panorama, which includes a panoramic topology layer and a perspective refinement layer; performing abnormal event location and perspective refinement based on the fused panorama to determine abnormal location data; determining a spatial distribution field based on the distributed camera gimbal, performing spatial field solution on the abnormal location data, and determining a target camera gimbal, wherein the target camera gimbal is a camera group with the abnormal location position as the monitoring range; controlling the target camera gimbal, performing refined monitoring based on the abnormal location data, and determining an abnormal location single column.

[0006] Another aspect disclosed in the present application provides an anomaly positioning system based on panoramic video fusion, the system comprising: an acquisition and return unit, the acquisition and return unit being used to acquire and return discrete video streams of the control area based on a distributed camera gimbal, and determine multiple video streams, wherein the multiple video streams are timestamp-identified and synchronized based on a network time protocol; a conversion and clustering unit, the conversion and clustering unit being used to perform coaxial conversion and homologous clustering on the multiple video streams, and determine a preprocessed video stream, wherein each cluster corresponds to the same local video stream and is marked with a relative spatial position; a fusion and splicing modeling unit, the fusion and splicing modeling unit being used to construct a three-dimensional panoramic fusion module to process the preprocessed video streams. Process the video stream for fusion and splicing modeling to determine the fused panorama, which includes a panoramic topology layer and a perspective refinement layer; a positioning and refinement unit, which is used to locate abnormal events and refine the perspective based on the fused panorama, and determine abnormal positioning data; a spatial field solution unit, which is used to determine the spatial distribution field based on the distributed camera gimbal, perform spatial field solution on the abnormal positioning data, and determine the target camera gimbal, which is a camera group with the abnormal positioning position as the monitoring range; a refined monitoring unit, which is used to control the target camera gimbal, perform refined monitoring based on the abnormal positioning data, and determine the abnormal positioning single column.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] The aforementioned anomaly location method based on panoramic video fusion uses a camera pan / tilt system (PTZ) to capture video streams and ensures they are precisely synchronized using timestamps and network time protocols. These multiple video streams are then processed and clustered according to their spatial location to form preprocessed video streams. This ensures that the video streams within each cluster originate from the same local area. These preprocessed video streams are then fused and spliced ​​using 3D panoramic fusion technology to construct a panoramic surveillance image. This panoramic image not only captures the topology of the entire area but also provides detailed, granular view of the area, enabling monitoring personnel to gain a comprehensive and detailed understanding of the entire area. Once an anomaly is detected in the panoramic image, it is immediately located and granularized to a specific viewpoint, generating anomaly location data. Based on this anomaly location data and the spatial distribution of the distributed camera pan / tilt systems, the optimal target camera pan / tilt system for monitoring the anomaly is calculated. Finally, these target camera pan / tilt systems are controlled to provide detailed monitoring of the anomaly and generate a more accurate anomaly location list. In this way, monitoring personnel can quickly and accurately understand the details of abnormal events, and take appropriate measures to deal with them, thereby improving the efficiency of abnormal event reverse investigation.

[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 The figure is a flow chart of an anomaly location method based on panoramic video fusion in one embodiment.

[0012] Figure 2 The following is an architecture diagram of an anomaly localization system based on panoramic video fusion in one embodiment.

[0013] Explanation of the accompanying symbols: collection and return unit 1, conversion and clustering unit 2, fusion splicing modeling unit 3, positioning and refinement unit 4, spatial field solution unit 5, refinement monitoring unit 6. DETAILED DESCRIPTION

[0014] The embodiments of the present application provide an anomaly positioning method and system based on panoramic video fusion to solve the technical problems of limited monitoring range and low positioning accuracy in traditional anomaly positioning methods.

[0015] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0016] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or modules that are not clearly listed or are inherent to these processes, methods, products or devices.

[0017] Example 1

[0018] like Figure 1As shown, the present application provides an anomaly positioning method based on panoramic video fusion, the method comprising:

[0019] Based on the distributed camera gimbal, discrete video streams are collected and transmitted back to the control area to determine multiple video streams. The multiple video streams are marked with timestamps and synchronized based on the network time protocol.

[0020] In an embodiment of the present application, the system terminal collects video of the target area by deploying camera pan-tilt platforms at multiple locations, and transmits these dispersed video streams back to the system terminal. These video streams are timestamped to ensure that the acquisition time of each video clip can be accurately identified in subsequent processing. In order to maintain the synchronization of the video streams, the system terminal pre-calibrates the time of all camera pan-tilt platforms using the Network Time Protocol (NTP) to ensure that they operate under the same time reference. In this way, even if multiple camera pan-tilt platforms are distributed in different physical locations, their video streams can be consistent in time, providing a reliable data basis for subsequent monitoring and analysis.

[0021] Coaxial conversion and homologous clustering are performed on the multiple video streams to determine a pre-processed video stream, wherein each cluster corresponds to the same local video stream and is marked with a relative spatial position.

[0022] In one embodiment, after acquiring multiple synchronized, time-stamped video streams, the system terminal performs coaxial conversion and homologous clustering on these streams to optimize their quality. Specifically, the system terminal uses the timestamps corresponding to each video stream to identify co-frequency video streams. These co-frequency video streams are then transformed using the world coordinate system as the primary coordinate system. The system terminal then clusters the conversion results, grouping video streams with the same local area to form multiple clusters. These clusters collectively constitute the preprocessed video stream. The video streams within each cluster in the preprocessed video stream are considered to be of the same origin, meaning they originate from a similar local area and contain similar information. Grouping related video streams together enables more efficient data processing and analysis. Furthermore, to more accurately represent the relationships between these video streams, the system terminal identifies the relative spatial position of each cluster. This helps determine the specific area covered by each video stream and provides important spatial information for subsequent panoramic fusion and anomaly localization.

[0023] Furthermore, the present application provides coaxial conversion and homologous clustering of the multi-channel video streams, including:

[0024] Based on the timestamp, determine the same-frequency video stream based on the multiple video streams; determine the acquisition coordinate system of the multiple video streams, take the world coordinate system as the main coordinate system, perform coordinate transformation on the same-frequency video stream, and determine the transformed video stream; based on the acquisition domain, perform clustering processing on the transformed video stream to determine the preprocessed video stream.

[0025] Preferably, the system terminal identifies video streams captured within the same time period, known as co-frequency video streams, based on the timestamp information in the video streams. These video streams may originate from different camera pan / tilt platforms, but they are synchronized in time and can therefore be correlated. Subsequently, the acquisition coordinate system for each camera pan / tilt platform is determined. These acquisition coordinate systems describe the viewing angle of the corresponding camera. Using these acquisition coordinate systems, the system terminal can obtain the coordinates corresponding to all co-frequency video streams. Next, a world coordinate system is established. The world coordinate system is a unified, global coordinate system that describes the entire monitoring area. The origin, axis, and unit length of the world coordinate system are set based on actual needs, typically selecting the center of the monitoring area as the origin. Next, intrinsic calibration is performed on each camera pan / tilt platform to determine its internal parameters, such as focal length and distortion coefficient. These parameters describe the mapping relationship between the camera coordinate system and the image coordinate system. Extrinsic calibration is then performed to determine the position and orientation of each camera pan / tilt platform in the world coordinate system. This is achieved by photographing a calibration object of known shape (such as a calibration plate), and by comparing the position of the calibration object in the image coordinate system and the world coordinate system, the extrinsic parameter matrix of the camera gimbal is solved. For the same-frequency video stream, the system terminal uses the parameters obtained by the internal and external parameter calibration to convert their coordinates in their respective acquisition coordinate systems to the world coordinate system. This involves coordinate transformation operations such as rotation and translation. After the coordinate transformation, each frame of the same-frequency video stream is converted to a unified world coordinate system, forming a transformed video stream.

[0026] After completing the coordinate transformation, the system terminal determines the monitoring range and field of view of each camera gimbal. These ranges constitute the acquisition domain of each camera gimbal. Keyframes are then extracted from these transformed video streams and the spatial coordinate information of objects in these keyframes is analyzed. Subsequently, based on the characteristics of the keyframes and application requirements, the system terminal sets ε and MinPts. ε refers to the neighborhood size, which defines the maximum distance between points within the neighborhood. MinPts is the minimum number of points required to qualify as a core point within the ε neighborhood. Each keyframe is then treated as a data point and the DBSCAN algorithm is applied to group the data points into clusters based on the ε and MinPts parameters. Because keyframes in the same cluster originate from similar acquisition domains, the system terminal groups the video streams containing the keyframes in these clusters into the same preprocessed video stream. These preprocessed video streams not only contain time synchronization information but also undergo coordinate transformation and clustering, making them more spatially continuous and consistent.

[0027] A three-dimensional panoramic fusion module is constructed to perform fusion and splicing modeling on the pre-processed video streams to determine a fused panorama, which includes a panoramic topology layer and a viewpoint refinement layer.

[0028] In one embodiment, the system terminal pre-builds a 3D panoramic fusion module based on a 3D fusion algorithm. This module is used to accurately fuse and stitch preprocessed video streams to form a complete panoramic view. Within the 3D panoramic fusion module, the system terminal traverses the preprocessed video streams and determines the correspondence between them and spatial locations. It then extracts features from multiple preprocessed video streams at the same spatial location, i.e., multi-channel video features, and applies the 3D fusion algorithm within the 3D panoramic fusion module to perform feature fusion. Based on the feature fusion results, the system terminal constructs a panoramic topology layer and a viewpoint refinement layer, and associates them through inter-layer mapping to produce a fused panorama. This panoramic image not only captures the macroscopic structure and layout of the entire scene, but also preserves detailed information from each viewpoint. This panoramic image provides a comprehensive perspective and rich scene details, further improving the efficiency of abnormal event investigation.

[0029] Furthermore, the present application provides a method for performing fusion and splicing modeling on the pre-processed video stream to determine a fusion panorama, and the method further includes:

[0030] The preprocessed video stream is traversed, and spatial position mapping is performed on each cluster to determine a position mapping relationship, wherein each spatial position corresponds to at least one video; based on the position mapping relationship, multi-channel video features of the preprocessed video stream at the same mapping position are extracted, and feature fusion is performed to determine a fusion feature; based on the fusion feature, three-dimensional modeling and spatial position-based splicing processing are performed to determine the fused panorama.

[0031] Preferably, in the 3D panoramic fusion module, the system terminal traverses the preprocessed video streams and spatially maps the multiple video streams within each cluster. This involves mapping these video streams to their corresponding spatial regions and determining a positional mapping relationship. In this relationship, each spatial location corresponds to one or more video streams. Subsequently, based on the resulting positional mapping relationship, the system terminal extracts multiple video features with the same mapping location from the preprocessed video streams. These features, such as color, texture, and shape, represent key information within the video. The system terminal then uses the 3D fusion algorithm within the 3D panoramic fusion module to fuse these features from different videos. The 3D fusion algorithm assigns a weight to each video stream based on its quality, importance, and historical experience. The multiple video features with the same mapping location are then weighted averaged according to the assigned weights to determine a fused feature. This fused feature comprehensively considers information from multiple video channels and balances the different video streams based on the weights. Based on the resulting fused features, the system terminal performs 3D modeling and spatially position-based splicing. By mapping these fused features into 3D space, a local model is constructed. These local models are then spliced ​​together to construct a panoramic topology layer. After obtaining the panoramic topology layer, the system terminal performs panoramic layer refinement on the panoramic topology layer to determine the viewpoint refinement layer. Finally, the panoramic topology layer and the viewpoint refinement layer are mapped and associated to determine the fused panorama.

[0032] Furthermore, the present application provides that after performing three-dimensional modeling and splicing processing based on spatial position, the method further includes:

[0033] Based on the fusion features, three-dimensional modeling is performed to determine a local model. Based on the spatial relative position, the local models are spatially spliced ​​to determine a panoramic model, which is stored in the panoramic topology layer. Panoramic layer refinement is performed to determine a multi-layer model, which is stored in the perspective refinement layer, wherein the layer is refined to the smallest unit. Inter-layer mapping associates the panoramic topology layer with the perspective refinement layer to determine the fused panorama.

[0034] Optionally, after obtaining the fused features, the system terminal uses the 3D modeling component to outline the overall shape and structure of the multiple objects represented by the fused features, establishing multiple base models. These base models are then refined and adjusted, including adjusting their size, shape, texture, material, and color, to achieve greater precision and a truer representation of reality. Subsequently, various details and features, such as patterns, textures, wrinkles, and bumps, are added to these models based on modeling requirements to enhance their realism. After adding these details and features, the system terminal completes the construction of the local models. Based on the determined position mapping relationships, the system terminal determines the relative position and orientation of each local model in 3D space. Using the stitching tool in the 3D modeling component, the local models are stitched together according to their relative positions and orientations in the real environment to form a complete panoramic model. This panoramic model is then encapsulated and stored in the pre-built panoramic topology layer structure, forming the final panoramic topology layer.

[0035] After obtaining the panoramic topology layer, the system terminal determines the number of refinement levels required based on the complexity and detail requirements of the panoramic model within the panoramic topology layer. For example, multiple refinement levels can be set, ranging from global to local, such as overall scene, main area, sub-area, and detail element. Subsequently, at each refinement level, the smallest unit to be refined is determined. These units can be objects, areas, textures, and so on within the scene, depending on the content of the panoramic model and actual requirements. Next, starting from the overall panoramic model, the model is divided into several major areas or sections. For example, in a factory, the first level of refinement might include main corridors, production workshops, and office areas. Then, further refinement is performed on the major areas or sections within each layer, breaking them down into smaller sub-areas or elements. Each level focuses on smaller details and more specific elements, such as different production lines in a production workshop or individual offices in an office area. At the final level, refinement is performed to the smallest unit, which can be a specific object, texture, label, etc. For example, an oil temperature meter or converter transformer cooler on a production line. Finally, the system terminal uses these refined multi-layer structures as a multi-layer model, and encapsulates and stores them in the pre-built view refinement layer structure to form the final view refinement layer.

[0036] After obtaining the panoramic topology layer and the viewpoint refinement layer, the system terminal associates them through inter-layer mapping. This association ensures that when the system terminal views the model from different viewpoints or at different levels, it can switch seamlessly and that the content seen is coherent and consistent. This association ultimately results in a fused panorama that presents both a comprehensive overview of the entire scene and in-depth observation of local details. This achieves full coverage, high-precision monitoring and positioning of the controlled area, and improves the efficiency of retrospective investigation of abnormal events.

[0037] Based on the fused panorama, abnormal event positioning and perspective refinement are performed to determine abnormal positioning data.

[0038] In one embodiment, when an abnormal event occurs, such as equipment failure or a safety incident, the system terminal uses the panoramic topology layer to analyze and fuse real-time data in the panorama to fuzzily locate the abnormal event. Once the approximate location of the abnormal event is determined, the system terminal calls the view refinement layer to refine the view to obtain more detailed abnormality information. During the view refinement process, the system terminal performs mapping and abnormality determination based on the view refinement layer to more accurately locate the abnormality. By combining the panoramic topology layer and the view refinement layer, the system terminal can obtain abnormality location data. This abnormality location data describes the specific circumstances of the abnormality and the specific location where the abnormality occurred, further improving the efficiency and accuracy of abnormality management and providing strong support for ensuring safe and stable production.

[0039] Furthermore, the present application provides a method for determining abnormal location data, which further includes:

[0040] Based on the fused panorama, combined with the panoramic topology layer, fuzzy positioning of abnormal events is performed to determine fuzzy positioning data; based on the fuzzy positioning data, mapping and abnormality judgment are performed in the perspective refinement layer to determine the abnormal positioning data, wherein the abnormal positioning data includes abnormal features-abnormal positions.

[0041] Preferably, the system terminal first parses the fused panorama to obtain real-time panoramic data. Subsequently, the panoramic topology layer is used to monitor the real-time panoramic data to understand the current status of each area. After that, it is determined whether the current status data of each area deviates from the normal status data, and the areas where the deviation is greater than or equal to the normal deviation threshold are marked. This normal deviation threshold is set based on historical experience and safety requirements. Then, the system terminal analyzes all areas with abnormal markings to determine whether there are adjacent abnormal areas. If so, these areas are fused. This is because when abnormalities occur in adjacent areas, it may be because an abnormality occurs in one area, resulting in a chain reaction. Finally, the fused abnormal areas and the unfused areas are sorted to determine the fuzzy positioning data.

[0042] After obtaining fuzzy positioning data, the system terminal transmits this data to the view refinement layer. At this layer, the system terminal maps the abnormal areas in the fuzzy positioning data within a multi-layer model and extracts the data corresponding to the minimum unit models in the mapped areas. Subsequently, using the same method described above, the system terminal determines the deviation between the data corresponding to the minimum unit models and normal data, identifying the abnormal minimum unit models. The devices corresponding to these minimum unit models are the source of the regional anomaly. The system terminal then extracts real-time data from the abnormal minimum unit models, obtains anomaly features, and determines the spatial location of the minimum unit models based on multiple models. The obtained anomaly features and spatial location are then combined to generate anomaly location data. This anomaly location method, combining the panoramic topology layer and the view refinement layer, not only improves the efficiency and accuracy of anomaly detection but also provides the system terminal with rich anomaly information, helping to better locate and handle anomaly events.

[0043] Determine the spatial distribution field based on the distributed camera pan-tilt platform, perform spatial field solution on the abnormal positioning data, and determine the target camera pan-tilt platform, where the target camera pan-tilt platform is a camera group with the abnormal positioning position as the monitoring range.

[0044] In one embodiment, the system terminal first determines the spatial distribution field of distributed camera pans. This means the system terminal needs to know the precise location of each camera pan in three-dimensional space and the monitoring area it covers. Once the anomaly location data is determined, the system terminal uses this information to solve the spatial distribution field. Specifically, the system terminal compares the location of the anomaly with the monitoring range of each camera pan in the spatial distribution field to identify those camera pans that cover the area where the anomaly occurred. During this solution process, the system terminal considers multiple factors, such as the viewing angle and focal length of the camera pan, to ensure that the selected camera pan can provide sufficient detail and clarity to observe and analyze the anomaly. Through the solution, a group of target camera pans is determined, which will be responsible for monitoring the area containing the anomaly location. These target camera pans may be one or more, depending on the nature and scale of the anomaly and the monitoring capabilities of the camera pan.

[0045] Furthermore, the present application provides a method for determining a target camera gimbal, the method further comprising:

[0046] Based on the spatial distribution field, the abnormal positioning data is screened for camera pans in the position space to determine a neighboring camera pan; based on full-view monitoring, the neighboring camera pans are screened to determine the target camera pan.

[0047] Preferably, when the system terminal obtains anomaly location data, it will screen camera pans close to the anomaly location based on the spatial distribution field and use these camera pans as neighboring camera pans. Because these camera pans are close to the anomaly point, they all have the prerequisites for accurately monitoring the abnormal device. The system terminal then screens these neighboring camera pans. The screening criteria is to ensure that the selected camera pans can provide full-view monitoring, that is, they can cover the area where the anomaly occurs as completely as possible. Through this step, the system terminal can identify one or more target camera pans that are not only close to the anomaly point but also provide the best monitoring angle.

[0048] Based on timing synchronization, the target camera pan / tilt platform is subjected to parameter control settings based on panoramic monitoring to determine time zone monitoring data; and mapping of the time zone monitoring data to the target camera pan / tilt platform is performed.

[0049] Preferably, after identifying the target camera pan / tilts, the system terminal performs parameter settings for these target camera pan / tilts based on panoramic monitoring, using time synchronization as a benchmark to obtain more comprehensive and accurate surveillance data. This parameter setting primarily involves adjusting the camera pan / tilt's pitch and horizontal angles to ensure a wider and clearer surveillance image. The pitch angle is the angle at which the pan / tilt rotates up and down; adjusting it changes the camera's viewing angle, allowing it to capture objects at different heights. The horizontal angle is the angle at which the pan / tilt rotates left and right; adjusting it changes the camera's horizontal viewing angle, expanding the surveillance range. By performing these parameter settings for the target camera pan / tilts, the system terminal can acquire surveillance data from different time zones, including images and videos captured from multiple angles and time points. The system terminal then maps these time zone surveillance data with the target camera pan / tilts, ensuring that each piece of data corresponds to the corresponding camera pan / tilt and monitoring time. This mapping not only facilitates subsequent data analysis and processing but also allows for rapid location of specific surveillance images when needed, providing powerful support for security monitoring and incident investigation.

[0050] The target camera pan / tilt is controlled to perform detailed monitoring based on the abnormal positioning data to determine the abnormal positioning column.

[0051] In one embodiment, after identifying target camera pan / tilts, the system terminal controls these target camera pan / tilts to perform detailed monitoring of the devices indicated by the anomaly location data. During this detailed monitoring process, the system terminal fine-tunes the target camera pan / tilts' shooting angles and focal lengths based on the anomaly location data to more accurately capture and record the details of the devices causing regional anomalies. Through detailed monitoring, the system terminal can further understand the anomaly patterns and severity of these devices, thereby more accurately determining the anomaly location column. This column, containing anomaly location information, anomaly pattern information, and anomaly severity information, is the core component of the anomaly event and is crucial for subsequent processing.

[0052] Furthermore, the present application provides that after determining the abnormal positioning column, the method further includes:

[0053] An abnormal event level assessment and evolution trend prediction are performed on the abnormal location single column to determine the abnormal assessment result; an early warning is issued for the abnormal assessment result, and an abnormal operation and maintenance single column is generated.

[0054] Preferably, after determining the anomaly location column, the system terminal extracts anomaly pattern information and anomaly severity information from the anomaly location column. Subsequently, the anomaly severity information is compared against a pre-configured grading table to match the corresponding abnormal event level. This grading table is pre-constructed based on historical data and expert advice. The anomaly severity information corresponding to each anomaly pattern is then time-seriesized, arranging it chronologically to ensure that each time point has corresponding anomaly severity information. An anomaly severity curve is then constructed based on the time-series data, with the abscissa representing time and the ordinate representing the anomaly severity. The system terminal then extracts multiple historical anomaly severity curves corresponding to the anomaly pattern from the anomaly database. The difference between the anomaly severity of each historical anomaly severity curve and the corresponding time point of the anomaly severity curve is calculated, and the corresponding mean is then calculated to measure the degree of deviation between each historical anomaly severity curve and the anomaly severity curve. After all deviations are calculated, the system terminal uses the development trend of the historical anomaly severity curve with the smallest deviation as the evolution trend prediction result for the anomaly location column. Finally, the system terminal uses the assessed abnormal event level and the evolution trend prediction result as the anomaly assessment result. Once the abnormality assessment results are obtained, the system terminal generates corresponding warning information based on the assessment results and generates an abnormal operation and maintenance list. This operation and maintenance list contains detailed information about the abnormal event and recommended response measures, providing clear guidance and reference for operation and maintenance personnel.

[0055] Determine the operation and maintenance management personnel based on the abnormal location data, integrate the abnormal location list, the abnormal assessment result and the abnormal operation and maintenance list, and send them to the terminal device of the operation and maintenance management personnel; among them, there is autonomous investigation based on personnel management authority and spontaneous investigation and transmission of the system.

[0056] Preferably, the system terminal determines which operation and maintenance managers are responsible for handling these anomalies based on the anomaly location data. This involves a clear understanding of the division of labor and responsibilities of the operation and maintenance team. Once the responsible personnel are determined, the system terminal integrates all information related to the anomaly, including an anomaly location column, an anomaly assessment result, and an anomaly operation and maintenance column. Subsequently, the integrated information is sent to the terminal device of the responsible operation and maintenance manager. In this way, the operation and maintenance manager can view this information on his own device and conduct anomaly investigation and processing accordingly. In this process, there are two investigation methods: one is autonomous investigation based on personnel management authority, that is, the operation and maintenance manager actively conducts anomaly investigation and processing based on his own professional knowledge and experience, combined with the information provided by the system terminal; the other is spontaneous investigation and transmission by the system terminal, that is, the system terminal conducts preliminary analysis and judgment of the anomaly, and sends relevant information to the operation and maintenance manager in a timely manner to assist in anomaly processing.

[0057] In summary, the embodiments of the present application have at least the following technical effects:

[0058] This embodiment of the present application uses distributed camera pan-tilt systems to collect discrete video streams within a controlled area and synchronizes them using the Network Time Protocol. Subsequently, the collected multi-channel video streams undergo coaxial conversion and homologous clustering to form preprocessed video streams. Each cluster represents a video stream from the same local area and is annotated with its relative spatial position. A 3D panoramic fusion module is then constructed to perform fusion and splicing modeling on the preprocessed video streams, generating a fused panorama consisting of a panoramic topology layer and a viewpoint refinement layer. Based on the fused panorama, abnormal events are located and the monitoring viewpoint is refined to determine abnormal location data. Based on this abnormal location data, the spatial distribution field of the distributed camera pan-tilt systems is used to determine target camera pan-tilt systems (i.e., the camera group that covers the abnormal location). These target camera pan-tilt systems are then controlled to perform refined monitoring based on the abnormal location data, generating an abnormal location list. Once the abnormal location list is obtained, the abnormal event level is assessed and its evolution trend is predicted to generate an abnormality assessment result. Based on the assessment results, an early warning is issued, generating an abnormal operation and maintenance list. Finally, the responsible operations and maintenance personnel are identified based on the anomaly location data. The anomaly location list, anomaly assessment results, and anomaly operation and maintenance list are integrated and sent to the relevant personnel's terminal devices. These technical effects collectively address the technical issues of limited monitoring range and low positioning accuracy in traditional anomaly location methods. This enables full coverage and high-precision monitoring and positioning of the controlled area through panoramic video fusion, improving the efficiency of abnormal event investigation.

[0059] Example 2

[0060] Based on the same inventive concept as the anomaly positioning method based on panoramic video fusion in the aforementioned embodiment, Figure 2 As shown, the present application provides an anomaly positioning system based on panoramic video fusion, the system comprising:

[0061] Capture and return unit 1: The capture and return unit 1 is used to capture and return discrete video streams of the control area based on a distributed camera pan / tilt platform, and determine multiple video streams, wherein the multiple video streams are timestamped and synchronized based on the network time protocol;

[0062] Conversion and clustering unit 2: The conversion and clustering unit 2 is used to perform coaxial conversion and homologous clustering on the multiple video streams to determine a pre-processed video stream, wherein each cluster corresponds to the same local video stream and is marked with a relative spatial position;

[0063] Fusion and splicing modeling unit 3: The fusion and splicing modeling unit 3 is used to construct a three-dimensional panoramic fusion module, perform fusion and splicing modeling on the pre-processed video stream, and determine a fused panorama, which includes a panoramic topology layer and a view refinement layer;

[0064] Positioning and refinement unit 4: The positioning and refinement unit 4 is used to perform abnormal event positioning and perspective refinement based on the fused panorama to determine abnormality positioning data;

[0065] Spatial field solving unit 5: The spatial field solving unit 5 is used to determine the spatial distribution field based on the distributed camera pan-tilt platform, perform spatial field solving on the abnormal positioning data, and determine the target camera pan-tilt platform, wherein the target camera pan-tilt platform is a camera group with the abnormal positioning position as the monitoring range;

[0066] Refined monitoring unit 6: The refined monitoring unit 6 is used to control the target camera pan / tilt platform, perform refined monitoring based on the abnormal positioning data, and determine the abnormal positioning column.

[0067] Furthermore, the conversion and clustering unit 2 is further configured to perform the following method:

[0068] Based on the timestamp, determine the same-frequency video stream based on the multiple video streams; determine the acquisition coordinate system of the multiple video streams, take the world coordinate system as the main coordinate system, perform coordinate transformation on the multiple video streams, and determine the transformed video stream; based on the acquisition domain, cluster the transformed video stream to determine the preprocessed video stream.

[0069] Furthermore, the fusion splicing modeling unit 3 is also used to perform the following method:

[0070] The preprocessed video stream is traversed, and spatial position mapping is performed on each cluster to determine a position mapping relationship, wherein each spatial position corresponds to at least one video; based on the position mapping relationship, multi-channel video features of the preprocessed video stream at the same mapping position are extracted, and feature fusion is performed to determine a fusion feature; based on the fusion feature, three-dimensional modeling and spatial position-based splicing processing are performed to determine the fused panorama.

[0071] Furthermore, the fusion splicing modeling unit 3 is also used to perform the following method:

[0072] Based on the fusion features, three-dimensional modeling is performed to determine a local model. Based on the spatial relative position, the local models are spatially spliced ​​to determine a panoramic model, which is stored in the panoramic topology layer. Panoramic layer refinement is performed to determine a multi-layer model, which is stored in the perspective refinement layer, wherein the layer is refined to the smallest unit. Inter-layer mapping associates the panoramic topology layer with the perspective refinement layer to determine the fused panorama.

[0073] Furthermore, the positioning and refinement unit 4 is further configured to perform the following method:

[0074] Based on the fused panorama, combined with the panoramic topology layer, fuzzy positioning of abnormal events is performed to determine fuzzy positioning data; based on the fuzzy positioning data, mapping and abnormality judgment are performed in the perspective refinement layer to determine the abnormal positioning data, wherein the abnormal positioning data includes abnormal features-abnormal positions.

[0075] Furthermore, the spatial field solving unit 5 is further configured to execute the following method:

[0076] Based on the spatial distribution field, the abnormal positioning data is screened for camera pan-tilts in the position space to determine the neighboring camera pan-tilts; based on full-view monitoring, the neighboring camera pan-tilts are screened to determine the target camera pan-tilts; based on time synchronization, the target camera pan-tilts are set for parameter control based on panoramic monitoring to determine the time zone monitoring data; and the time zone monitoring data is mapped to the target camera pan-tilts.

[0077] Furthermore, the detailed monitoring unit 6 is further configured to execute the following method:

[0078] The abnormal event level and evolution trend of the abnormal location list are evaluated and predicted to determine the abnormal assessment result; an early warning is issued for the abnormal assessment result, and an abnormal operation and maintenance list is generated; an operation and maintenance manager based on the abnormal location data is determined, and the abnormal location list, the abnormal assessment result and the abnormal operation and maintenance list are integrated and sent to the terminal device of the operation and maintenance manager; among them, there is autonomous investigation based on personnel management authority and spontaneous investigation and transmission of the system.

[0079] It should be noted that the order in which the embodiments of the present application are presented is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. The processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0081] This specification and drawings are merely illustrative of the present application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Obviously, those skilled in the art may make various modifications and variations to this application without departing from the scope of this application. Thus, this application is intended to include such modifications and variations as fall within the scope of this application and its equivalents.

Claims

1. An anomaly location method based on panoramic video fusion, characterized in that: The method comprises: Based on the distributed camera pan-tilt system, discrete video streams are collected and transmitted back to the control area to determine multiple video streams. The multiple video streams are timestamped and synchronized based on the network time protocol. Performing coaxial conversion and homologous clustering on the multiple video streams to determine a pre-processed video stream, wherein each cluster corresponds to a same local video stream and is marked with a relative spatial position; Constructing a 3D panoramic fusion module to perform fusion and splicing modeling on the pre-processed video streams and determine a fused panorama, which includes a panoramic topology layer and a viewpoint refinement layer; Based on the fused panorama, abnormal event positioning and perspective refinement are performed to determine abnormal positioning data; Determine a spatial distribution field based on the distributed camera pan / tilt platform, perform spatial field solution on the abnormal positioning data, and determine a target camera pan / tilt platform, where the target camera pan / tilt platform is a camera group with the abnormal positioning position as a monitoring range; Controlling the target camera pan / tilt platform to perform detailed monitoring based on the abnormal positioning data and determine the abnormal positioning column; Performing coaxial conversion and homologous clustering on the multiple video streams includes: Determining a same-frequency video stream based on the multiple video streams based on the timestamp; Determine the acquisition coordinate system of the multiple video streams, take the world coordinate system as the main coordinate system, perform coordinate conversion on the same-frequency video streams, and determine the converted video streams; Based on the acquisition domain, clustering processing is performed on the converted video stream to determine the pre-processed video stream; Performing fusion and splicing modeling on the pre-processed video stream to determine a fusion panorama, including: Traversing the pre-processed video stream, performing intra-cluster spatial position mapping on each cluster, and determining a position mapping relationship, wherein each spatial position corresponds to at least one video; Based on the position mapping relationship, extracting multi-channel video features of the same mapping position of the pre-processed video stream, and performing feature fusion to determine a fusion feature; Based on the fusion features, three-dimensional modeling and splicing processing based on spatial positions are performed to determine the fused panorama; After 3D modeling and splicing based on spatial position, it includes: Based on the fusion features, three-dimensional modeling is performed to determine a local model; based on the spatial relative positions, the local models are spatially spliced ​​to determine a panoramic model, and the panoramic model is stored in the panoramic topology layer; Performing panoramic layer refinement to determine a multi-layer model and storing it in the perspective refinement layer, wherein the layer refinement is performed to the smallest unit; Inter-layer mapping associates the panoramic topology layer with the view refinement layer to determine the fused panorama.

2. The anomaly location method based on panoramic video fusion according to claim 1, characterized in that: The determining of abnormal location data includes: Based on the fused panorama, combined with the panoramic topology layer, fuzzy positioning of abnormal events is performed to determine fuzzy positioning data; Based on the fuzzy positioning data, mapping and abnormality determination are performed at the view refinement layer to determine the abnormality positioning data, wherein the abnormality positioning data includes abnormality features-abnormal positions.

3. The anomaly location method based on panoramic video fusion according to claim 1, characterized in that: Determining the target camera gimbal includes: Based on the spatial distribution field, the abnormal positioning data is screened for camera pan / tilts in a position space to determine a neighboring camera pan / tilt; Based on full-view monitoring, the neighboring camera pan / tilts are screened to determine the target camera pan / tilt; Based on the timing synchronization, the target camera gimbal is set with parameters and controls under panoramic monitoring to determine the time zone monitoring data; Mapping the time zone monitoring data to the target camera pan / tilt platform is performed.

4. The anomaly location method based on panoramic video fusion according to claim 1, characterized in that: After determining the abnormal location column, the method includes: Performing abnormal event level assessment and evolution trend prediction on the abnormal location column to determine the abnormality assessment result; Issue an early warning for the abnormal assessment results and generate a single abnormal operation and maintenance column; Determine an operation and maintenance manager based on the abnormality location data, integrate the abnormality location list, the abnormality assessment result, and the abnormal operation and maintenance list, and send them to a terminal device of the operation and maintenance manager; Among them, there are autonomous inspections based on personnel management authority and spontaneous inspections and transmissions by the system.

5. An anomaly positioning system based on panoramic video fusion, characterized in that: The steps for implementing the anomaly positioning method based on panoramic video fusion as described in any one of claims 1 to 4 include: Collection and return unit: Based on the distributed camera pan-tilt platform, it collects and returns discrete video streams of the control area, determines multiple video streams, wherein the multiple video streams are timestamped and synchronized based on the network time protocol; A conversion and clustering unit: performing coaxial conversion and homologous clustering on the multiple video streams to determine a pre-processed video stream, wherein each cluster corresponds to a local video stream and is marked with a relative spatial position; Fusion and splicing modeling unit: constructing a three-dimensional panoramic fusion module, performing fusion and splicing modeling on the pre-processed video stream, and determining a fused panorama, which includes a panoramic topology layer and a view refinement layer; Positioning and refinement unit: Based on the fused panorama, perform abnormal event positioning and perspective refinement to determine abnormal positioning data; A spatial field solving unit is configured to determine a spatial distribution field based on the distributed camera pan-tilt platform, perform spatial field solving on the abnormal positioning data, and determine a target camera pan-tilt platform, wherein the target camera pan-tilt platform is a camera group with the abnormal positioning position as a monitoring range; Refined monitoring unit: controls the target camera pan / tilt platform, performs refined monitoring based on the abnormal positioning data, and determines the abnormal positioning column.

Images