Abnormality positioning method and system based on panoramic video fusion

Abstract

The invention relates to the technical field of anomaly positioning, and provides an anomaly positioning method and system based on panoramic video fusion. The method comprises the following steps: carrying out discrete video stream acquisition and return on a management and control area, and determining multiple paths of video streams; converting and clustering the multiple paths of video streams, and determining a preprocessed video stream; constructing a three-dimensional panorama fusion module, performing fusion splicing modeling on the preprocessed video stream, and determining a fusion panorama; carrying out abnormal event positioning and visual angle refinement based on the fused panorama, and determining abnormal positioning data; determining a spatial distribution field, performing spatial field calculation for the abnormal positioning data, and determining a target camera holder; and controlling the target camera holder, carrying out refined monitoring, and determining an abnormal positioning single column. According to the method, the technical problems of limited monitoring range and low positioning precision in a traditional anomaly positioning method are solved, full-coverage and high-precision monitoring and positioning of the control area through panoramic video fusion are realized, and the effect of improving the abnormal event reverse checking efficiency is achieved.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, specifically to the field of abnormality positioning technology, and in particular to an abnormality positioning method and system based on panoramic video fusion. Background Art

[0002] With the rapid development of industrial production and the improvement of automation, the timely detection and location of equipment failures and major emergency events in industrial environments have become crucial. These events may not only lead to production interruptions, but also have serious impacts on personnel safety and the environment. However, the traditional industrial abnormality location method adopts a combination of manual regular inspections and monitoring equipment. However, this model has problems such as low positioning accuracy and slow response speed due to human factors and the existence of monitoring blind spots, which makes it difficult to meet the needs of modern industrial production for efficiency, accuracy and safety. Summary of the invention

[0003] The present application provides an anomaly positioning method and system based on panoramic video fusion, aiming to solve the technical problems of limited monitoring range and low positioning accuracy in traditional anomaly positioning methods.

[0004] In view of the above problems, the present application provides an anomaly positioning method and system based on panoramic video fusion.

[0005] The first aspect disclosed in the present application provides an abnormality positioning method based on panoramic video fusion, the method comprising: based on a distributed camera gimbal, collecting and returning discrete video streams of a control area, determining multiple video streams, wherein the multiple video streams are timestamped and synchronized based on a network time protocol; coaxial conversion and homologous clustering of the multiple video streams to determine a preprocessed video stream, wherein each cluster corresponds to a local video stream and is marked with a spatial relative position; constructing a three-dimensional panoramic fusion module, fusion splicing modeling of the preprocessed video stream, and determining a fused panorama, which includes a panoramic topology layer and a view refinement layer; based on the fused panorama, abnormal event positioning and view refinement are performed to determine abnormal positioning data; determining a spatial distribution field based on the distributed camera gimbal, performing spatial field solution for the abnormal positioning data, and determining a target camera gimbal, wherein the target camera gimbal is a camera group with an abnormal positioning position as a monitoring range; controlling the target camera gimbal, performing refined monitoring based on the abnormal positioning data, and determining an abnormal positioning single column.

[0006] Another aspect disclosed in the present application provides an anomaly positioning system based on panoramic video fusion, the system comprising: an acquisition and return unit, the acquisition and return unit is used to acquire and return discrete video streams of the control area based on a distributed camera gimbal, and determine multiple video streams, wherein the multiple video streams are marked with timestamps and synchronized based on a network time protocol; a conversion and clustering unit, the conversion and clustering unit is used to perform coaxial conversion and homologous clustering on the multiple video streams, and determine a pre-processed video stream, wherein each cluster corresponds to the same local video stream and is marked with a relative spatial position; a fusion splicing modeling unit, the fusion splicing modeling unit is used to construct a three-dimensional panoramic fusion module to perform a pre-processing on the pre-processed video stream. Process the video stream for fusion and splicing modeling to determine the fused panorama, which includes a panoramic topology layer and a view refinement layer; a positioning and refinement unit, the positioning and refinement unit is used to locate abnormal events and refine the view based on the fused panorama, and determine abnormal positioning data; a spatial field solution unit, the spatial field solution unit is used to determine the spatial distribution field based on the distributed camera gimbal, perform spatial field solution on the abnormal positioning data, and determine the target camera gimbal, the target camera gimbal is a camera group with the abnormal positioning position as the monitoring range; a refined monitoring unit, the refined monitoring unit is used to control the target camera gimbal, perform refined monitoring based on the abnormal positioning data, and determine the abnormal positioning single column.

[0007] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0008] The above-mentioned anomaly positioning method based on panoramic video fusion collects video streams through camera pan-tilts and ensures that these video streams are accurately synchronized through timestamps and network time protocols. Subsequently, these multi-channel video streams are processed and clustered according to their spatial position relationship to form pre-processed video streams, so that the video streams in each clustering cluster come from the same local area. After that, the three-dimensional panoramic fusion technology is used to fuse and splice these pre-processed video streams to construct a panoramic monitoring screen. This panoramic screen not only contains the topological structure of the entire area, but also can be refined to a specific perspective, so that the monitoring personnel can fully and carefully understand the situation of the entire area. Once an abnormal event is found in the panoramic screen, it will be immediately located and refined to a specific perspective to generate abnormal positioning data. Then, based on these abnormal positioning data, combined with the spatial distribution field of the distributed camera pan-tilt, the target camera pan-tilt group that is most suitable for monitoring these abnormal events is calculated. Finally, these target camera pan-tilts are controlled to perform detailed monitoring of abnormal events and generate more accurate abnormal positioning columns. In this way, monitoring personnel can quickly and accurately understand the details of abnormal events, and take appropriate measures to deal with them, thereby improving the efficiency of reverse investigation of abnormal events.

[0009] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 The figure is a flow chart of an abnormality location method based on panoramic video fusion in one embodiment.

[0012] Figure 2 The following is an architecture diagram of an abnormality localization system based on panoramic video fusion in one embodiment.

[0013] Explanation of the accompanying drawings: collection and feedback unit 1, conversion and clustering unit 2, fusion splicing modeling unit 3, positioning and refinement unit 4, spatial field solution unit 5, refinement monitoring unit 6. DETAILED DESCRIPTION

[0014] The embodiments of the present application provide an anomaly positioning method and system based on panoramic video fusion to solve the technical problems of limited monitoring range and low positioning accuracy in traditional anomaly positioning methods.

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0016] It should be noted that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or modules that are not explicitly listed or inherent to these processes, methods, products or devices.

[0017] Embodiment 1

[0018] like Figure 1As shown, the present application provides an abnormality positioning method based on panoramic video fusion, the method comprising:

[0019] Based on the distributed camera gimbal, discrete video streams are collected and transmitted back to the control area to determine multiple video streams, wherein the multiple video streams are marked with timestamps and synchronized based on the network time protocol.

[0020] In an embodiment of the present application, the system terminal collects video of the target area by deploying camera pan-tilts at multiple locations, and transmits these scattered video streams back to the system terminal. These video streams are all timestamped to ensure that the acquisition time of each video clip can be accurately identified in subsequent processing. In order to maintain the synchronization of the video streams, the system terminal pre-calibrates the time of all camera pan-tilts using the Network Time Protocol (NTP) to ensure that they work under the same time reference. In this way, even if multiple camera pan-tilts are distributed in different physical locations, their video streams can be consistent in time, providing a reliable data basis for subsequent monitoring and analysis.

[0021] Coaxial conversion and homologous clustering are performed on the multiple video streams to determine a pre-processed video stream, wherein each cluster corresponds to a local video stream and is marked with a relative spatial position.

[0022] In one embodiment, after acquiring multiple video streams with timestamps and synchronization, the system terminal performs coaxial conversion and homologous clustering on these multiple video streams to optimize the quality of these multiple video streams. Specifically, the system terminal determines the same-frequency video stream using the timestamp corresponding to each multiple video stream. Subsequently, the world coordinate system is used as the main coordinate system to perform coordinate conversion on these determined same-frequency video streams. Afterwards, the system terminal clusters the conversion results, clusters the video streams with the same local area, and forms multiple cluster clusters, which together constitute the preprocessed video stream. The video streams in each cluster cluster in the preprocessed video stream are regarded as homologous, that is, they come from similar local areas and contain similar information. In this way, by classifying related video streams together, the data can be processed and analyzed more efficiently. In addition, in order to more accurately represent the relationship between these video streams, the system terminal identifies the spatial relative position of each cluster cluster. This helps to determine the specific area covered by each video stream and provides important spatial information in the subsequent panoramic fusion and abnormal positioning process.

[0023] Furthermore, the present application provides coaxial conversion and homologous clustering of the multi-channel video streams, including:

[0024] Based on the timestamp, determine the same-frequency video stream based on the multiple video streams; determine the acquisition coordinate system of the multiple video streams, take the world coordinate system as the main coordinate system, perform coordinate transformation on the same-frequency video stream, and determine the transformed video stream; based on the acquisition domain, perform clustering processing on the transformed video stream to determine the preprocessed video stream.

[0025] Preferably, the system terminal identifies those video streams collected in the same time period, i.e., the same-frequency video streams, based on the timestamp information in the video stream. These video streams may come from different camera pan platforms, but they are synchronized in time and can therefore be associated with each other. Subsequently, the acquisition coordinate system of each camera pan platform is determined, and these acquisition coordinate systems are used to describe the viewing angle of the corresponding camera. Through these acquisition coordinate systems, the system terminal can obtain the coordinates corresponding to all the same-frequency video streams. After that, a world coordinate system is established, which is a unified, global coordinate system used to describe the entire monitoring area. The origin, axial direction, and unit length of the world coordinate system are set according to actual needs, and the center of the monitoring area is usually selected as the origin. Then, each camera pan platform is internally calibrated to determine its internal parameters, such as focal length, distortion coefficient, etc. These parameters describe the mapping relationship from the camera coordinate system to the image coordinate system. Then, external calibration is performed to determine the position and direction of each camera pan platform in the world coordinate system. This is achieved by photographing a calibration object of known shape (such as a calibration plate), and by comparing the position of the calibration object in the image coordinate system and the world coordinate system, the external parameter matrix of the camera gimbal is solved. For the same-frequency video stream, the system terminal uses the parameters obtained by the internal and external calibration to convert their coordinates in their respective acquisition coordinate systems to the world coordinate system. This involves coordinate transformation operations such as rotation and translation. After the coordinate transformation, each frame of the same-frequency video stream is converted to a unified world coordinate system to form a transformed video stream.

[0026] After completing the coordinate conversion, the system terminal determines the monitoring range and field of view of each camera pan / tilt, which constitute the acquisition domain of each camera pan / tilt. Then, key frames are extracted from these converted video streams, and the spatial coordinate information of objects in these key frames is analyzed. Subsequently, based on the characteristics of the key frames and the needs of the application, the system terminal sets ε and MinPts. Among them, ε refers to the neighborhood size. It defines the maximum distance of points in the neighborhood. MinPts refers to the minimum number of points required to become a core point in the ε neighborhood. After that, each key frame is regarded as a data point, and the DBSCAN algorithm is applied to divide the data points into different clusters based on the ε and MinPts parameters. Because the key frames in the same cluster come from similar acquisition domains, the system terminal classifies the video streams to which the key frames in these clusters belong as the same preprocessed video stream. These preprocessed video streams not only contain time synchronization information, but also undergo coordinate conversion and clustering processing, making them more continuous and consistent in space.

[0027] A three-dimensional panoramic fusion module is constructed to perform fusion and splicing modeling on the pre-processed video streams to determine a fused panorama, which includes a panoramic topology layer and a viewing angle refinement layer.

[0028] In one embodiment, the system terminal pre-constructs a three-dimensional panoramic fusion module based on a three-dimensional fusion algorithm. This module is used to accurately fuse and splice the pre-processed video streams to form a complete panoramic view. Inside the three-dimensional panoramic fusion module, the system terminal traverses the pre-processed video streams to determine the correspondence between the pre-processed video streams and the spatial position. Subsequently, multiple pre-processed video streams of the same spatial position, i.e., multi-channel video features, are extracted, and the three-dimensional fusion algorithm inside the three-dimensional panoramic fusion module is called for feature fusion. Afterwards, based on the feature fusion results, the system terminal constructs a panoramic topology layer and a view refinement layer, and associates the panoramic topology layer with the view refinement layer by inter-layer mapping to obtain a fused panorama. This panoramic image not only contains the macroscopic structure and layout of the entire scene, but also retains the detailed information at each view. Through this panoramic image, an all-round view and rich scene detail information are obtained, further improving the efficiency of abnormal event reverse checking.

[0029] Furthermore, the present application provides a method for performing fusion splicing modeling on the preprocessed video stream to determine a fusion panorama, and the method further includes:

[0030] The preprocessed video stream is traversed, and spatial position mapping is performed on each cluster cluster to determine the position mapping relationship, wherein each spatial position corresponds to at least one video; based on the position mapping relationship, multi-channel video features of the preprocessed video stream with the same mapping position are extracted, and feature fusion is performed to determine the fusion feature; based on the fusion feature, three-dimensional modeling and spatial position-based stitching processing are performed to determine the fused panorama.

[0031] Preferably, in the three-dimensional panoramic fusion module, the system terminal traverses the pre-processed video stream and performs spatial position mapping on the multiple video streams in each cluster, that is, the video streams are matched with the spatial regions to which they belong, and the position mapping relationship is determined. In this relationship, each spatial position corresponds to one or more video streams. Subsequently, based on the obtained position mapping relationship, the system terminal extracts multiple video features with the same mapping position from the pre-processed video stream. These features include color, texture, shape, etc., which represent key information in the video. Afterwards, the system terminal calls the three-dimensional fusion algorithm inside the three-dimensional panoramic fusion module to fuse these features from different videos. The three-dimensional fusion algorithm assigns a weight to each video stream according to the quality, importance, and historical experience of the video stream. Then, the multi-channel video features with the same mapping position are weighted averaged according to the assigned weights to determine the fusion feature. This fusion feature comprehensively considers the information of multiple videos and weighs different video streams according to the weights. Then, based on the obtained fusion feature, the system terminal performs three-dimensional modeling and splicing processing based on spatial position. By mapping these fusion features into three-dimensional space, a local model is constructed. These local models are then spliced ​​together to construct a panoramic topology layer. After obtaining the panoramic topology layer, the system terminal performs panoramic layer refinement on the panoramic topology layer to determine the view refinement layer. Finally, the panoramic topology layer and the view refinement layer are associated with each other through inter-layer mapping to determine the fused panorama.

[0032] Furthermore, the present application provides that after performing three-dimensional modeling and splicing processing based on spatial position, the method also includes:

[0033] Based on the fusion features, three-dimensional modeling is performed to determine the local model. Based on the spatial relative position, the local model is spatially spliced ​​to determine the panoramic model, which is stored in the panoramic topology layer. Panoramic layer refinement is performed to determine a multi-layer model, which is stored in the perspective refinement layer, wherein the layer is refined to the smallest unit. Inter-layer mapping associates the panoramic topology layer with the perspective refinement layer to determine the fused panorama.

[0034] Optionally, after obtaining the fusion features, the system terminal outlines the overall shape and structure of multiple objects represented by the fusion features in the three-dimensional modeling component, and establishes multiple basic models. Subsequently, these basic models are refined and adjusted, including adjusting the size, shape, texture, material, color, etc. of the model to make it more refined and close to the actual situation. After that, according to the modeling requirements, various details and features are added to these models, such as patterns, textures, wrinkles, bumps, etc., to increase the realism of the model. After the details and features are added, the system terminal completes the construction of the local model. Then, according to the determined position mapping relationship, the system terminal clarifies the relative position and direction of each local model in the three-dimensional space. Then, using the splicing tool in the three-dimensional modeling component, the local models are spliced ​​according to their relative positions and directions in the actual environment to form a complete panoramic model, and the panoramic model is encapsulated and stored in the pre-built panoramic topology layer structure to form the final panoramic topology layer.

[0035] After obtaining the panoramic topology layer, the system terminal sets the number of layers to be refined according to the complexity and detail requirements of the panoramic model inside the panoramic topology layer. For example, multiple refinement levels can be set from the whole to the part, such as the overall scene, main area, sub-area, detail element, etc. Then, at each refinement level, the smallest unit to be refined is determined. These units can be objects, areas, textures, etc. in the scene, depending on the content of the panoramic model and actual needs. After that, starting from the overall panoramic model, the model is divided into several main areas or parts. For example, in a factory, the first level of refinement may include the main channel, production workshop, office area, etc. Then, for the main area or part in each layer, continue to refine it and split it into smaller sub-areas or elements. Each level focuses on smaller details and more specific elements. For example, different production lines in the production workshop, individual offices in the office area, etc. At the final level, refine to the smallest unit, which may be a specific object, texture, label, etc. For example, the oil temperature meter on the production line, the converter cooler, etc. Finally, the system terminal uses these refined multi-layer structures as multi-layer models, and encapsulates and stores them in the pre-built view refinement layer structure to form the final view refinement layer.

[0036] After obtaining the panoramic topology layer and the view refinement layer, the system terminal associates the panoramic topology layer and the view refinement layer through inter-layer mapping. This association ensures that when the system terminal views the model at different viewpoints or different levels, it can switch seamlessly, and the content seen is coherent and consistent. Through this association, a fused panorama is finally obtained, which can not only show the overall picture of the entire scene, but also provide in-depth observation of local details. It realizes full coverage, high-precision monitoring and positioning of the control area, and improves the efficiency of abnormal event reverse investigation.

[0037] Based on the fused panorama, abnormal event positioning and viewing angle refinement are performed to determine abnormal positioning data.

[0038] In one embodiment, when an abnormal event occurs, such as equipment failure, safety accident, etc., the system terminal uses the panoramic topology layer to analyze and fuse the real-time data in the panorama to fuzzily locate the abnormal event. Once the approximate location of the abnormal event is determined, the system terminal calls the view refinement layer to refine the view to obtain more detailed abnormal information. During the view refinement process, the system terminal performs mapping and abnormality judgment according to the view refinement layer to more accurately locate the abnormality. Through the joint use of the panoramic topology layer and the view refinement layer, the system terminal can obtain abnormal location data. This abnormal location data describes the specific situation of the abnormality and the specific location where the abnormality occurs, so as to further improve the efficiency and accuracy of abnormal management and provide strong support for ensuring the safety and stability of production.

[0039] Furthermore, the present application provides a method for determining abnormal location data, and the method further includes:

[0040] Based on the fused panorama, the abnormal event is fuzzy located in combination with the panoramic topology layer to determine the fuzzy location data; based on the fuzzy location data, mapping and abnormality judgment are performed in the perspective refinement layer to determine the abnormal location data, wherein the abnormal location data includes abnormal features-abnormal positions.

[0041] Preferably, the system terminal first parses the fused panorama to obtain real-time panoramic data. Subsequently, the panoramic topology layer is used to monitor the real-time panoramic data to understand the current status of each area. After that, it is determined whether the current status data of each area deviates from the normal status data, and the areas where the deviation is greater than or equal to the normal deviation threshold are marked. This normal deviation threshold is set based on historical experience and safety requirements. Then, the system terminal analyzes all areas with abnormal markings to determine whether there are adjacent abnormal areas. If so, these areas are fused. This is because when abnormalities occur in adjacent areas, it may be because an abnormality occurs in one area, resulting in a chain reaction. Finally, the fused abnormal areas and the unfused areas are sorted to determine the fuzzy positioning data.

[0042] After obtaining the fuzzy positioning data, the system terminal transmits the obtained fuzzy positioning data to the perspective refinement layer. In this layer, the system terminal maps the abnormal area in the fuzzy positioning data in the multi-layer model and extracts the data corresponding to the minimum unit model in the mapped area. Subsequently, the same method as above is used to determine the deviation between the data corresponding to the minimum unit model and the normal data, and identify the minimum unit model with abnormalities. The devices corresponding to these minimum unit models are the source of regional abnormalities. After that, the system terminal extracts the real-time data of the minimum unit model with abnormalities, obtains abnormal features, and determines the spatial position of the minimum unit model based on multiple models. Then, the obtained abnormal features and spatial positions are combined to form abnormal positioning data. The abnormal positioning method combining the panoramic topology layer and the perspective refinement layer not only improves the efficiency and accuracy of abnormal detection, but also provides the system terminal with rich abnormal information, which helps to better locate and handle abnormal events.

[0043] Determine the spatial distribution field based on the distributed camera gimbal, perform spatial field solution on the abnormal positioning data, and determine the target camera gimbal, where the target camera gimbal is a camera group with the abnormal positioning position as the monitoring range.

[0044] In one embodiment, the system terminal first determines the spatial distribution field of the distributed camera pan-tilts. This means that the system terminal needs to know the exact location of each camera pan-tilt in three-dimensional space and the monitoring area they cover. When the abnormal location data is determined, the system terminal uses this information to solve the spatial distribution field. Specifically, the system terminal compares the location of the abnormal location with the monitoring range of each camera pan-tilt in the spatial distribution field to find those camera pan-tilts that can cover the area where the abnormality occurs. In this solution process, the system terminal will consider multiple factors, such as the viewing angle and focal length of the camera pan-tilt, to ensure that the selected camera pan-tilt can provide sufficient details and clarity to observe and analyze abnormal events. Through the solution, a group of target camera pan-tilts can be determined, which will be responsible for monitoring the area containing the abnormal location. These target camera pan-tilts may be one or more, and the specific number depends on the nature and scale of the abnormal event and the monitoring capability of the camera pan-tilt.

[0045] Furthermore, the present application provides a method for determining a target camera gimbal, the method further comprising:

[0046] Based on the spatial distribution field, the camera pan-tilts in the position space of the abnormal positioning data are screened to determine the neighborhood camera pan-tilts; based on full-view monitoring, the neighborhood camera pan-tilts are screened to determine the target camera pan-tilts.

[0047] Preferably, when the system terminal obtains the abnormal location data, it will screen the camera pan-tilts close to the abnormal position based on the spatial distribution field, and use these camera pan-tilts as neighborhood camera pan-tilts. Because these camera pan-tilts are close to the abnormal point, they all have the premise of accurately monitoring the abnormal equipment. Subsequently, the system terminal screens these neighborhood camera pan-tilts. The screening criterion is to ensure that the selected camera pan-tilt can provide full-view monitoring, that is, it can cover the area where the abnormality occurs as completely as possible. Through this step, the system terminal can determine one or more target camera pan-tilts, which are not only close to the abnormal point, but also can provide the best monitoring angle.

[0048] Based on the timing synchronization, the target camera pan-tilt platform is subjected to parameter control setting based on the panoramic monitoring, and the time zone monitoring data is determined; and the time zone monitoring data is mapped with the target camera pan-tilt platform.

[0049] Preferably, after determining the target camera pan-tilt, in order to obtain more comprehensive and accurate monitoring data, the system terminal will use timing synchronization as a reference to perform parameter control settings on these target camera pan-tilts under panoramic monitoring. The parameter control settings mainly include adjusting the pitch angle and horizontal angle of the camera pan-tilt to ensure that a wider and clearer monitoring screen can be captured. The pitch angle is the angle at which the camera pan-tilt rotates up and down. Adjusting it can change the camera's viewing angle height, thereby photographing targets at different heights. The horizontal angle is the angle at which the camera pan-tilt rotates left and right. Adjusting the horizontal angle can change the camera's horizontal viewing angle, making the monitoring range wider. By performing such parameter control settings on the target camera pan-tilt, the system terminal can obtain monitoring data in different time zones, which include images and videos captured from multiple angles and time points. Subsequently, the system terminal maps these time zone monitoring data with the target camera pan-tilt to ensure that each piece of data corresponds to the corresponding camera pan-tilt and monitoring time. Such a mapping relationship not only helps with subsequent data analysis and processing, but also can quickly locate specific monitoring screens when needed, providing strong support for security monitoring and abnormal event reverse checking.

[0050] The target camera pan / tilt is controlled to perform detailed monitoring based on the abnormal positioning data to determine the abnormal positioning single column.

[0051] In one embodiment, after the target camera pan-tilts are determined, the system terminal will control these target camera pan-tilts to perform detailed monitoring on the devices pointed to by the abnormal location data. During the detailed monitoring process, the system terminal will fine-tune the shooting angle and focal length of the target camera pan-tilts according to the abnormal location data, so as to more accurately capture and record the details of the devices that cause regional abnormalities. Through detailed monitoring, the system terminal can further understand the abnormal patterns and abnormality levels of these devices, so as to more accurately determine the abnormal location single column. This single column contains abnormal location information, abnormal pattern information and abnormality level information. It is the core part of the abnormal event and is crucial for subsequent processing.

[0052] Furthermore, the present application provides that after determining the abnormal positioning column, the method further includes:

[0053] The abnormal event level assessment and evolution trend prediction are performed on the abnormal location list to determine the abnormal assessment result; an early warning is issued for the abnormal assessment result, and an abnormal operation and maintenance list is generated.

[0054] Preferably, after determining the abnormal location single column, the system terminal extracts the abnormal mode information and abnormal degree information from the abnormal location single column. Subsequently, based on the set rating table, the abnormal degree information is compared to match the corresponding abnormal event level. This rating table is pre-built based on historical data and expert advice. Then the abnormal degree information corresponding to each abnormal mode information is time-series, that is, arranged according to the time before and after, to ensure that each time point has corresponding abnormal degree information. After that, an abnormal degree curve is constructed based on the time-series data, and the abscissa of the abnormal degree curve is time and the ordinate is the abnormal degree. Then, the system terminal extracts multiple historical abnormal degree curves corresponding to the abnormal mode from the abnormal database, calculates the abnormal degree difference between each historical abnormal degree curve and the abnormal degree curve corresponding to the time point, and then calculates the corresponding mean value, which is used to measure the deviation degree between each historical abnormal degree curve and the abnormal degree curve. After all the deviation degrees are calculated, the system terminal uses the development trend of the historical abnormal degree curve with the smallest deviation as the evolution trend prediction result of the abnormal location single column. Finally, the system terminal uses the evaluated abnormal event level and the evolution trend prediction result as the abnormal evaluation result. Once the abnormal assessment results are obtained, the system terminal generates corresponding warning information based on the assessment results and generates an abnormal operation and maintenance list. This operation and maintenance list contains detailed information on the abnormal event and recommended response measures, etc., providing clear guidance and reference for operation and maintenance personnel.

[0055] Determine the operation and maintenance management personnel based on the abnormal location data, integrate the abnormal location list, the abnormal assessment result and the abnormal operation and maintenance list, and send them to the terminal device of the operation and maintenance management personnel; among them, there is autonomous investigation based on personnel management authority and spontaneous investigation and transmission of the system.

[0056] Preferably, the system terminal determines which operation and maintenance managers are responsible for handling these exceptions based on the exception location data. This involves a clear understanding of the personnel division of labor and responsibilities of the operation and maintenance team. Once the responsible personnel are determined, the system terminal integrates all information related to the exception, including the exception location list, the exception assessment results, and the exception operation and maintenance list. Subsequently, the integrated information is sent to the terminal device of the responsible operation and maintenance manager. In this way, the operation and maintenance manager can view this information on his own device and conduct exception investigation and processing accordingly. In this process, there are two troubleshooting methods: one is autonomous troubleshooting based on personnel management authority, that is, the operation and maintenance manager actively conducts exception investigation and processing based on his own professional knowledge and experience, combined with the information provided by the system terminal; the other is the spontaneous troubleshooting and transmission of the system terminal, that is, the system terminal conducts a preliminary analysis and judgment of the exception, and sends the relevant information to the operation and maintenance manager in a timely manner to assist in exception processing.

[0057] In summary, the embodiments of the present application have at least the following technical effects:

[0058] The embodiment of the present application collects discrete video streams in the control area through a distributed camera pan-tilt, and synchronizes time through a network time protocol. Subsequently, the collected multi-channel video streams are coaxially converted and homologous clustered to form a pre-processed video stream, each cluster represents a video stream in the same local area, and is marked with a relative spatial position. After that, a three-dimensional panoramic fusion module is constructed to fuse and splice the pre-processed video streams to generate a fused panorama including a panoramic topology layer and a view refinement layer. On the basis of the fused panorama, the abnormal event is located, and the monitoring view is refined to determine the abnormal positioning data. Then, based on the abnormal positioning data, the spatial distribution field of the distributed camera pan-tilt is used to determine the target camera pan-tilt through spatial field solution, that is, the camera group that can cover the abnormal positioning position. Then, these target camera pan-tilts are controlled to perform refined monitoring based on the abnormal positioning data, and an abnormal positioning single column is generated. After obtaining the abnormal positioning single column, the abnormal event level is evaluated and the evolution trend is predicted to form an abnormal evaluation result. And according to the evaluation results, an early warning is issued to generate an abnormal operation and maintenance single column. Finally, the responsible operation and maintenance manager is determined based on the abnormal location data, and the abnormal location list, abnormal assessment results, and abnormal operation and maintenance list are integrated and sent to the terminal devices of relevant personnel. These technical effects jointly solve the technical problems of limited monitoring range and low positioning accuracy in traditional abnormal location methods, and achieve full coverage and high-precision monitoring and positioning of the control area through panoramic video fusion, thereby improving the efficiency of abnormal event reverse investigation.

[0059] Embodiment 2

[0060] Based on the same inventive concept as the anomaly positioning method based on panoramic video fusion in the aforementioned embodiment, Figure 2 As shown, the present application provides an abnormality positioning system based on panoramic video fusion, the system comprising:

[0061] Collection and return unit 1: The collection and return unit 1 is used to collect and return discrete video streams of the control area based on a distributed camera pan / tilt, and determine multiple video streams, wherein the multiple video streams are marked with timestamps and synchronized based on a network time protocol;

[0062] Conversion and clustering unit 2: the conversion and clustering unit 2 is used to perform coaxial conversion and homologous clustering on the multi-channel video streams to determine the pre-processed video stream, wherein each cluster corresponds to the same local video stream and is marked with a relative spatial position;

[0063] Fusion splicing modeling unit 3: The fusion splicing modeling unit 3 is used to construct a three-dimensional panoramic fusion module, perform fusion splicing modeling on the pre-processed video stream, and determine a fused panorama, which includes a panoramic topology layer and a viewing angle refinement layer;

[0064] Positioning and refinement unit 4: the positioning and refinement unit 4 is used to perform abnormal event positioning and viewpoint refinement based on the fused panorama, and determine abnormal positioning data;

[0065] Spatial field solving unit 5: the spatial field solving unit 5 is used to determine the spatial distribution field based on the distributed camera pan-tilt platform, perform spatial field solving for the abnormal positioning data, and determine the target camera pan-tilt platform, wherein the target camera pan-tilt platform is a camera group with the abnormal positioning position as the monitoring range;

[0066] Refined monitoring unit 6: The refined monitoring unit 6 is used to control the target camera gimbal, perform refined monitoring based on the abnormal positioning data, and determine the abnormal positioning column.

[0067] Furthermore, the conversion and clustering unit 2 is also used to perform the following method:

[0068] Based on the timestamp, determine the same-frequency video stream based on the multiple video streams; determine the acquisition coordinate system of the multiple video streams, take the world coordinate system as the main coordinate system, perform coordinate transformation on the multiple video streams, and determine the transformed video stream; based on the acquisition domain, perform clustering processing on the transformed video stream to determine the preprocessed video stream.

[0069] Furthermore, the fusion splicing modeling unit 3 is also used to perform the following method:

[0070] The preprocessed video stream is traversed, and spatial position mapping is performed on each cluster cluster to determine the position mapping relationship, wherein each spatial position corresponds to at least one video; based on the position mapping relationship, multi-channel video features of the preprocessed video stream with the same mapping position are extracted, and feature fusion is performed to determine the fusion feature; based on the fusion feature, three-dimensional modeling and spatial position-based stitching processing are performed to determine the fused panorama.

[0071] Furthermore, the fusion splicing modeling unit 3 is also used to perform the following method:

[0072] Based on the fusion features, three-dimensional modeling is performed to determine the local model. Based on the spatial relative position, the local model is spatially spliced ​​to determine the panoramic model, which is stored in the panoramic topology layer. Panoramic layer refinement is performed to determine a multi-layer model, which is stored in the perspective refinement layer, wherein the layer is refined to the smallest unit. Inter-layer mapping associates the panoramic topology layer with the perspective refinement layer to determine the fused panorama.

[0073] Furthermore, the positioning and refinement unit 4 is also used to perform the following method:

[0074] Based on the fused panorama, the abnormal event is fuzzy located in combination with the panoramic topology layer to determine the fuzzy location data; based on the fuzzy location data, mapping and abnormality judgment are performed in the perspective refinement layer to determine the abnormal location data, wherein the abnormal location data includes abnormal features-abnormal positions.

[0075] Furthermore, the spatial field solving unit 5 is also used to execute the following method:

[0076] Based on the spatial distribution field, the abnormal positioning data is screened for camera pans in the position space to determine the neighborhood camera pan; based on full-view monitoring as a benchmark, the neighborhood camera pans are screened to determine the target camera pan; based on timing synchronization as a benchmark, the target camera pan is set for parameter control based on panoramic monitoring to determine the time zone monitoring data; and the time zone monitoring data is mapped with the target camera pan.

[0077] Furthermore, the detailed monitoring unit 6 is also used to execute the following method:

[0078] The abnormal event level and evolution trend of the abnormal location list are evaluated and predicted to determine the abnormal evaluation result; an early warning is issued for the abnormal evaluation result, and an abnormal operation and maintenance list is generated; the operation and maintenance management personnel based on the abnormal location data are determined, and the abnormal location list, the abnormal evaluation result and the abnormal operation and maintenance list are integrated and sent to the terminal device of the operation and maintenance management personnel; among them, there is autonomous investigation based on personnel management authority and spontaneous investigation and transmission of the system.

[0079] It should be noted that the above-mentioned sequence of the embodiments of the present application is only for description and does not represent the advantages and disadvantages of the embodiments. And the above-mentioned specific embodiments of this specification are described. The processes depicted in the accompanying drawings do not necessarily require the specific order and continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0080] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0081] This specification and drawings are merely exemplary illustrations of the present application and are deemed to cover any and all modifications, variations, combinations or equivalents within the scope of the present application. Obviously, a person skilled in the art may make various modifications and variations to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these modifications and variations.

Claims

1. An abnormality positioning method based on panoramic video fusion, characterized in that: The method comprises: Based on the distributed camera gimbal, discrete video streams are collected and transmitted back to the control area to determine multiple video streams, wherein the multiple video streams are marked with timestamps and synchronized based on the network time protocol; Performing coaxial conversion and homologous clustering on the multiple video streams to determine a preprocessed video stream, wherein each cluster corresponds to a local video stream and is marked with a relative spatial position; Constructing a three-dimensional panoramic fusion module, performing fusion and splicing modeling on the pre-processed video stream, and determining a fused panorama, which includes a panoramic topology layer and a viewing angle refinement layer; Based on the fused panorama, abnormal event positioning and viewing angle refinement are performed to determine abnormal positioning data; Determine a spatial distribution field based on the distributed camera pan-tilt platform, perform spatial field solution on the abnormal positioning data, and determine a target camera pan-tilt platform, wherein the target camera pan-tilt platform is a camera group with the abnormal positioning position as a monitoring range; The target camera pan / tilt is controlled to perform detailed monitoring based on the abnormal positioning data to determine the abnormal positioning single column.

2. The method for locating anomalies based on panoramic video fusion according to claim 1, characterized in that: The multi-channel video streams are coaxially converted and homologously clustered, including: Based on the timestamp, determining a same-frequency video stream based on the multiple video streams; Determine the acquisition coordinate system of the multiple video streams, take the world coordinate system as the main coordinate system, perform coordinate transformation on the same-frequency video streams, and determine the transformed video streams; Based on the acquisition domain, clustering processing is performed on the converted video stream to determine the pre-processed video stream.

3. The anomaly positioning method based on panoramic video fusion according to claim 1, characterized in that: The pre-processed video stream is subjected to fusion and splicing modeling to determine a fusion panorama, including: Traversing the preprocessed video stream, performing intra-cluster spatial position mapping for each cluster, and determining a position mapping relationship, wherein each spatial position corresponds to at least one video; Based on the position mapping relationship, extracting the multi-channel video features of the same mapping position of the pre-processed video stream, and performing feature fusion to determine the fusion feature; Based on the fusion features, three-dimensional modeling and splicing processing based on spatial positions are performed to determine the fused panorama.

4. The method for locating anomalies based on panoramic video fusion according to claim 3, characterized in that: After 3D modeling and splicing based on spatial position, it includes: Based on the fusion features, three-dimensional modeling is performed to determine a local model, and based on the spatial relative position, the local model is spatially spliced ​​to determine a panoramic model, which is stored in the panoramic topology layer; Performing panorama layer refinement to determine a multi-layer model and store it in the perspective refinement layer, wherein the layer refinement is performed to the smallest unit; Inter-layer mapping associates the panoramic topology layer with the view refinement layer to determine the fused panorama.

5. The anomaly positioning method based on panoramic video fusion according to claim 1, characterized in that: The determining of abnormal location data includes: Based on the fused panorama, combined with the panoramic topology layer, fuzzy positioning of abnormal events is performed to determine fuzzy positioning data; Based on the fuzzy positioning data, mapping and abnormality determination are performed at the viewing angle refinement layer to determine the abnormality positioning data, wherein the abnormality positioning data includes abnormality features-abnormal positions.

6. The anomaly positioning method based on panoramic video fusion according to claim 1, characterized in that: The step of determining a target camera gimbal comprises: Based on the spatial distribution field, the camera pan / tilt screening is performed on the abnormal positioning data in the position space to determine the neighboring camera pan / tilt; Based on full-view monitoring, the neighboring camera pan / tilts are screened to determine the target camera pan / tilt; Based on the timing synchronization, the target camera gimbal is set with parameters and controls based on the panoramic monitoring to determine the time zone monitoring data; Mapping the time zone monitoring data to the target camera pan / tilt is performed.

7. The method for locating anomalies based on panoramic video fusion according to claim 1, characterized in that: After determining the abnormal location column, the method includes: Performing abnormal event level assessment and evolution trend prediction on the abnormal location column to determine the abnormal assessment result; Issue an early warning for the abnormal evaluation result and generate an abnormal operation and maintenance list; Determine an operation and maintenance manager based on the abnormality location data, integrate the abnormality location list, the abnormality assessment result and the abnormal operation and maintenance list, and send them to a terminal device of the operation and maintenance manager; Among them, there are autonomous inspections based on personnel management authority, and spontaneous inspections and transmissions of the system.

8. An abnormality positioning system based on panoramic video fusion, characterized in that: The steps for implementing the method for locating anomalies based on panoramic video fusion as described in any one of claims 1 to 7 include: Collection and return unit: Based on the distributed camera pan / tilt, discrete video streams are collected and returned to the control area to determine multiple video streams, where the multiple video streams are marked with timestamps and synchronized based on the network time protocol; Conversion and clustering unit: performing coaxial conversion and homologous clustering on the multiple video streams to determine a pre-processed video stream, wherein each cluster corresponds to a local video stream and is marked with a relative spatial position; A fusion splicing modeling unit: constructs a three-dimensional panoramic fusion module, performs fusion splicing modeling on the pre-processed video stream, and determines a fused panorama, which includes a panoramic topology layer and a viewing angle refinement layer; Positioning and refinement unit: based on the fused panorama, abnormal event positioning and perspective refinement are performed to determine abnormal positioning data; A spatial field solving unit: determining a spatial distribution field based on the distributed camera pan-tilt platform, performing spatial field solving on the abnormal positioning data, and determining a target camera pan-tilt platform, wherein the target camera pan-tilt platform is a camera group with the abnormal positioning position as a monitoring range; Refined monitoring unit: controls the target camera gimbal to perform refined monitoring based on the abnormal positioning data and determine the abnormal positioning column.

Images