An event positioning method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide an event positioning method and device, electronic equipment and storage medium, relating to the technical field of visual positioning, comprising: acquiring a side-view panoramic visible light image of a to-be-detected scene area and a DEM side view, wherein the DEM side view comprises position information of each point in the to-be-detected scene area; matching the same scene points in the side-view panoramic visible light image and the DEM side view to obtain a scene point matching result; performing dense mapping on the side-view panoramic visible light image and the DEM side view according to the scene point matching result to obtain a dense mapping relationship; and determining the position information of each point in the side-view panoramic visible light image according to the position information of each point in the DEM side view and the dense mapping relationship. The embodiments of the present application improve the positioning information accuracy in a large scene.

Description

Technical Field

[0001] This application relates to the field of visual positioning technology, and in particular to an event positioning method, apparatus, electronic device and storage medium. Background Technology

[0002] Large, sparsely populated areas such as forests and valleys are difficult to locate. In the event of an emergency, achieving long-distance, high-precision monitoring and location of the incident to promptly resolve the crisis has always been a pressing challenge. For example, in the event of a fire or other emergency, pinpointing the location of the fire has always been a difficult aspect of fire suppression.

[0003] Most of the positioning methods proposed in related technologies have problems such as difficulty in implementation and low positioning accuracy. For example, the scheme of using high-point cameras installed on watchtowers to monitor the fire point of a large-scale fire using video still cannot accurately determine the location of the fire point due to geographical reasons. Summary of the Invention

[0004] The purpose of this application is to provide an event localization method, apparatus, electronic device, and storage medium to improve the accuracy of localization information in large-scale scenarios. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide an event location method, including:

[0006] The first side-view visible light image of the scene area to be detected is used to detect events of a preset type, and the event detection results are obtained.

[0007] If the event detection result indicates that a preset type of event exists in the first side-view visible light image, the first viewpoint information when the camera captures the first side-view visible light image and the first region in the first side-view visible light image where the preset type of event occurs are obtained;

[0008] Based on the first perspective information, determine the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected;

[0009] Based on the predetermined position information of each point in the side-view panoramic visible light image, the position information of the first mapped position is used as the location of a preset type event.

[0010] In one embodiment of this application, the method further includes:

[0011] Acquire at least one auxiliary image of the first side-view visible light image, and acquire second view information of the auxiliary image, wherein the acquisition field of view of the auxiliary image includes the field of view corresponding to the first region, and the magnification of the auxiliary image is less than the magnification of the first side-view visible light image;

[0012] Determining the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected based on the first viewpoint information includes:

[0013] Based on the first perspective information and the second perspective information, determine the second mapping position of the first region in the auxiliary image;

[0014] Based on the second perspective information and the second mapping position, the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected is determined.

[0015] In one embodiment of this application, the first perspective information is first PTZ information;

[0016] The step of acquiring at least one auxiliary image of the first side-view visible light image, and acquiring second-view information of the auxiliary image, includes:

[0017] N auxiliary images of the first side-view visible light image are obtained, and the PTZ information of the auxiliary images is obtained, wherein N is an integer greater than 1, the P and T information of the first side-view visible light image is the same as that of the first N-1 auxiliary images, the magnification of the N-1th auxiliary image is the same as that of the Nth auxiliary image, the magnification of the 1st auxiliary image is greater than that of the first side-view visible light image, the magnification of the (i+1)th auxiliary image is greater than that of the 1st auxiliary image, i∈[1, N-1], and i is an integer;

[0018] Determining the second mapping position of the first region in the auxiliary image based on the first viewpoint information and the second viewpoint information includes:

[0019] Based on the first PTZ information and the PTZ information of the first auxiliary image, determine the mapping position of the first region in the first auxiliary image;

[0020] Based on the PTZ information of the i-th and (i+1)-th auxiliary images, the mapping position of the first region in the (i+1)-th auxiliary image is determined, wherein the second mapping position is the mapping position of the first region in the N-th auxiliary image.

[0021] In one embodiment of this application, the process of pre-determining the position information of each point in the side-view panoramic visible light image includes:

[0022] Acquire a side-view panoramic visible light image of the scene area to be detected, and a DEM side view, wherein the DEM side view includes the position information of each point in the scene area to be detected;

[0023] Match the same scene points in the side-view panoramic visible light image and the DEM side view to obtain scene point matching results;

[0024] Based on the scene point matching results, dense mapping is performed on the side-view panoramic visible light image and the DEM side view to obtain a dense mapping relationship;

[0025] Based on the position information of each point in the DEM side view and the dense mapping relationship, the position information of each point in the side-view panoramic visible light image is determined.

[0026] In one embodiment of this application, acquiring the side-view panoramic visible light image of the scene region to be detected, and the DEM side view, includes:

[0027] Acquire sample side-view visible light images of the scene region to be detected from multiple different viewpoints using a visible light camera, where adjacent viewpoints have overlapping fields of view;

[0028] Based on the acquisition angle of the side-view visible light images of each sample, the side-view visible light images of each sample are stitched together in a panoramic view to obtain a side-view panoramic visible light image.

[0029] Obtain the DEM image of the scene area to be detected, wherein the DEM image includes the elevation information and latitude and longitude information of each point in the scene area to be detected;

[0030] A side view of the DEM is generated based on the elevation and latitude / longitude information of each point in the DEM image and the elevation and latitude / longitude information of the visible light camera.

[0031] In one embodiment of this application, the step of performing dense mapping on the side-view panoramic visible light image and the DEM side view based on the scene point matching result to obtain a dense mapping relationship includes:

[0032] The side-view panoramic visible light image is divided into multiple visible light regions according to the scene points, and the DEM side view is divided into multiple DEM side view regions.

[0033] Based on the scene point matching results, the mapping transformation matrix between the visible light region and the side view region of the DEM with matching relationship is calculated to obtain the dense mapping relationship.

[0034] Secondly, embodiments of this application provide an event location device, including:

[0035] The event detection module is used to detect preset types of events in the first side view visible light image of the scene area to be detected, and obtain the event detection results;

[0036] The first region acquisition is used to acquire, when the event detection result indicates that there is a preset type of event in the first side-view visible light image, the first view angle information when the camera captures the first side-view visible light image and the first region in the first side-view visible light image where the preset type of event occurs;

[0037] The first mapping position determination module is used to determine the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected, based on the first view information.

[0038] The event localization module is used to locate the first mapped position as a preset type event based on the pre-determined position information of each point in the side-view panoramic visible light image.

[0039] In one embodiment of this application, the apparatus further includes:

[0040] The image information acquisition module is used to acquire at least one auxiliary image of the first side-view visible light image, and to acquire second view information of the auxiliary image, wherein the acquisition field of view of the auxiliary image includes the field of view corresponding to the first region, and the magnification of the auxiliary image is less than the magnification of the first side-view visible light image.

[0041] The first mapping position determination module includes:

[0042] The second mapping position determination submodule is used to determine the second mapping position of the first region in the auxiliary image based on the first view information and the second view information;

[0043] The first mapping position determination submodule is used to determine the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected, based on the second view information and the second mapping position.

[0044] In one embodiment of this application, the first perspective information is first PTZ information;

[0045] The image information acquisition module is specifically used for:

[0046] N auxiliary images of the first side-view visible light image are obtained, and the PTZ information of the auxiliary images is obtained, wherein N is an integer greater than 1, the P and T information of the first side-view visible light image is the same as that of the first N-1 auxiliary images, the magnification of the N-1th auxiliary image is the same as that of the Nth auxiliary image, the magnification of the 1st auxiliary image is greater than that of the first side-view visible light image, the magnification of the (i+1)th auxiliary image is greater than that of the 1st auxiliary image, i∈[1, N-1], and i is an integer;

[0047] The second mapping position determination submodule is specifically used for:

[0048] Based on the first PTZ information and the PTZ information of the first auxiliary image, determine the mapping position of the first region in the first auxiliary image;

[0049] Based on the PTZ information of the i-th and (i+1)-th auxiliary images, the mapping position of the first region in the (i+1)-th auxiliary image is determined, wherein the second mapping position is the mapping position of the first region in the N-th auxiliary image.

[0050] In one embodiment of this application, the apparatus further includes:

[0051] The image acquisition module is used to acquire a side-view panoramic visible light image of the scene area to be detected, and a DEM side view, wherein the DEM side view includes the position information of each point in the scene area to be detected;

[0052] The scene point matching module is used to match the same scene points in the side-view panoramic visible light image and the DEM side view to obtain scene point matching results.

[0053] The dense mapping module is used to perform dense mapping on the side-view panoramic visible light image and the DEM side view based on the scene point matching results, so as to obtain a dense mapping relationship;

[0054] The location information determination module is used to determine the location information of each point in the side-view panoramic visible light image based on the location information of each point in the DEM side view and the dense mapping relationship.

[0055] In one embodiment of this application, the image acquisition module is specifically used for:

[0056] Acquire sample side-view visible light images of the scene region to be detected from multiple different viewpoints using a visible light camera, where adjacent viewpoints have overlapping fields of view;

[0057] Based on the acquisition angle of the side-view visible light images of each sample, the side-view visible light images of each sample are stitched together in a panoramic view to obtain a side-view panoramic visible light image.

[0058] Obtain the DEM image of the scene area to be detected, wherein the DEM image includes the elevation information and latitude and longitude information of each point in the scene area to be detected;

[0059] A side view of the DEM is generated based on the elevation and latitude / longitude information of each point in the DEM image and the elevation and latitude / longitude information of the visible light camera.

[0060] In one embodiment of this application, the dense mapping module is specifically used for:

[0061] The side-view panoramic visible light image is divided into multiple visible light regions according to the scene points, and the DEM side view is divided into multiple DEM side view regions.

[0062] Based on the scene point matching results, the mapping transformation matrix between the visible light region and the side view region of the DEM with matching relationship is calculated to obtain the dense mapping relationship.

[0063] Thirdly, embodiments of this application provide an electronic device, including:

[0064] Memory, used to store computer programs;

[0065] A processor, when executing a program stored in memory, implements any of the event location methods described in this application.

[0066] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the event location methods described in this application.

[0067] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute any of the event location methods described above.

[0068] Beneficial effects of the embodiments in this application:

[0069] The event localization method, apparatus, electronic device, and storage medium provided in this application detect preset type events in a first side-view visible light image of a scene area to be detected, obtaining an event detection result. If the event detection result indicates the presence of a preset type event in the first side-view visible light image, the method acquires first viewpoint information when the camera captures the first side-view visible light image and a first region in the first side-view visible light image where the preset type event occurs. Based on the first viewpoint information, the method determines a first mapping position of the first region in a side-view panoramic visible light image of the scene area to be detected. Based on the pre-determined position information of each point in the side-view panoramic visible light image, the method uses the position information of the first mapping position as the localization of the preset type event. This can improve the accuracy of localization information in large scenes.

[0070] Of course, implementing any product or method of this application does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0072] Figure 1-1 A flowchart illustrating a location information determination method provided in an embodiment of this application;

[0073] Figure 1-2 An example diagram of a DEM image provided in an embodiment of this application;

[0074] Figure 1-3 An example diagram of scene point matching provided in an embodiment of this application;

[0075] Figure 2-1 This is one possible implementation of step S101 provided in the embodiments of this application;

[0076] Figure 2-2 An example diagram of a single-view visible light image provided in an embodiment of this application;

[0077] Figure 2-3 An example diagram of panoramic visible light image stitching provided in this application embodiment;

[0078] Figure 2-4 An example diagram of a DEM side view provided for an embodiment of this application;

[0079] Figure 2-5 An example diagram of a side-view panoramic visible light image provided in an embodiment of this application;

[0080] Figure 3-1 This is one possible implementation of step S103 provided in the embodiments of this application;

[0081] Figure 3-2 An example diagram of a region division provided in an embodiment of this application;

[0082] Figure 3-3 A comparative example of a side-view panoramic visible light image and a densely mapped DEM side view is provided for embodiments of this application;

[0083] Figure 4 A flowchart illustrating an event localization method provided in an embodiment of this application;

[0084] Figure 5 A flowchart illustrating a location information determination method and an event location method provided in an embodiment of this application;

[0085] Figure 6 This is a schematic diagram of the structure of a location information determination device provided in an embodiment of this application;

[0086] Figure 7 This is a schematic diagram of the structure of an event location device provided in an embodiment of this application;

[0087] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0088] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of this application.

[0089] Since it is difficult to achieve accurate positioning in large-scale scenarios in related technologies, this application provides a method for determining location information and an event positioning method to solve this problem.

[0090] The following detailed description is provided through specific embodiments.

[0091] The location information determination method and event localization method provided in this application embodiment can be applied to any electronic device with computing capabilities, such as servers, cameras, hard disk recorders, personal computers, etc.

[0092] Firstly, such as Figure 1-1 As shown in the figure, this application provides a flowchart of a method for determining location information, including:

[0093] Step S101: Obtain a side-view panoramic visible light image of the scene area to be detected, as well as a side view of the DEM.

[0094] The DEM side view includes the location information of each point in the scene area to be detected.

[0095] The scene area to be monitored can be any scene area that needs to be monitored, such as forests, valleys, wetlands, nature reserves, etc.

[0096] A panoramic visible light image is a 360-degree optical image of the scene area to be inspected in the vertical direction. It can display the 360-degree view of the scene area in two-dimensional form through wide-angle representation. A side-view panoramic visible light image is a side view image of the panoramic mode of the scene area to be inspected. Specifically, the side-view panoramic visible light image of the scene area to be inspected can be acquired by any optical imaging device with panoramic image acquisition capabilities, or it can be obtained by stitching together multiple non-panoramic visible light images; there is no limitation here.

[0097] A DEM (Digital Elevation Model) can reflect local topographic features at a certain resolution and can be used to extract a large amount of surface morphology information. It is an important source of raw data for studying and analyzing topography, watersheds, and feature identification, for example... Figure 1-2 As shown, a DEM (Digital Elevation Model) is a bird's-eye view image. The image coordinates within it can be converted into GPS (Global Positioning System) coordinates using the transformation relationships stored within the DEM itself. The corresponding stored value is the elevation (height) of that point, thus enabling precise positioning of the scene within the DEM image in the real 3D world. Furthermore, the DEM also serves as fundamental data for a large scene area and can be obtained from any relevant DEM image acquisition method.

[0098] The DEM side view of the scene area to be detected is a side view image of the scene area in the form of DEM. The scene area corresponding to the DEM side view is the same as that of the side-view panoramic visible light image. Both are 360-degree views of the scene area to be detected. They are panoramic views of the scene area to be detected in the vertical direction. The DEM side view also includes the location information of each point in the scene area to be detected, such as the GPS coordinates and elevation of each point.

[0099] Step S102: Match the same scene points in the side-view panoramic visible light image and the DEM side view to obtain scene point matching results.

[0100] Matching identical scene points in the side-view panoramic visible light image and the DEM side view yields a one-to-one matching relationship between scene points in both images. Specifically, preliminary positioning can be achieved in the DEM side view based on obvious features of the scene region in the side-view panoramic visible light image, such as ridgeline trends. Then, based on the continuity of scene regions in both images, matching regions are determined, and precise matching is performed. Precise matching can be implemented using any common machine vision or deep learning-based methods and feature point matching algorithms, such as SIFT (Scale Invariant Feature Transform) and ORB (Oriented Fast and Rotated Brief). Manual recalibration can also be performed after algorithm calibration. Figure 1-3 As shown. The scene point matching result obtained at this time is a sparse mapping between the side-view panoramic visible light image and the DEM side view.

[0101] Step S103: Based on the scene point matching result, perform dense mapping on the side-view panoramic visible light image and the DEM side view to obtain a dense mapping relationship.

[0102] After obtaining the scene point matching results between the side-view panoramic visible light image and the DEM side view, a dense mapping is performed on the side-view panoramic visible light image and the DEM side view. This can be achieved using any common dense mapping algorithm, such as Dense-Sift (SIFT algorithm) or kNN (k-nearest neighbor classification) algorithm. This results in the dense mapping relationship between each scene point between the side-view panoramic visible light image and the DEM side view.

[0103] Step S104: Determine the position information of each point in the side-view panoramic visible light image based on the position information of each point in the DEM side view and the dense mapping relationship.

[0104] Based on the dense mapping relationship between the side-view panoramic visible light image and the DEM side view, each point in the DEM side view is mapped to each point in the side-view panoramic visible light image. Given the position information of each point in the DEM side view, the position information of each point in the side-view panoramic visible light image is determined based on the dense mapping relationship.

[0105] The location information determination method provided in this application has high positioning accuracy because it performs dense mapping between the side-view panoramic visible light image and the DEM side view. Even in a large scene area at a long distance, if the number of matching points under dense mapping meets a certain requirement, it can still have high positioning accuracy. Based on the positioning information obtained in this way, the accuracy of positioning information in large scenes is improved.

[0106] In one embodiment of this application, such as Figure 2-1 As shown, step S101 above acquires a side-view panoramic visible light image of the scene area to be detected, and a side view of the DEM, including:

[0107] Step S201: Acquire side-view visible light images of the scene area to be detected, captured by a visible light camera from multiple different perspectives.

[0108] In this case, adjacent viewpoints have overlapping fields of view.

[0109] Side-view visible light images of the scene area to be detected were acquired from multiple different perspectives, such as... Figure 2-2 As shown, there is a certain overlap area between adjacent viewpoints in the side-view visible light images of each sample.

[0110] In one example, the visible light camera is a gimbal camera. The visible light camera acquires side-view visible light images of the sample from multiple different perspectives by rotating the camera, while its own position remains unchanged.

[0111] Step S202: Based on the acquisition angle of the side-view visible light images of each sample, perform panoramic stitching on the side-view visible light images of each sample to obtain a side-view panoramic visible light image.

[0112] Based on the acquisition perspective of each sample's side-view visible light image, side-view visible light images of samples with overlapping fields of view are treated as adjacent images and stitched together panoramically to obtain a side-view panoramic visible light image, for example... Figure 2-3 As shown, the area within the box in the figure is Figure 2-2 Mid-scene area.

[0113] Step S203: Obtain the DEM image of the scene area to be detected.

[0114] The DEM image includes elevation and latitude / longitude information of each point in the scene area to be detected.

[0115] Step S204: Generate a DEM side view based on the elevation and latitude / longitude information of each point in the DEM image and the elevation and latitude / longitude information of the visible light camera.

[0116] The side view of the DEM of the scene region to be detected is obtained by converting the DEM image of the scene region to a side view from the camera's perspective, such as... Figure 2-4 As shown, Figure 1-2 The DEM side view of the mid-field attractions has a horizontal field of view covering 360 degrees. The corresponding coordinate values ​​can store GPS coordinates, elevation information, and converted XYZ coordinates in a Cartesian coordinate system with the camera as the origin. Figure 2-5 and Figure 2-3 The images are a side view (360-degree field of view) of the DEM and a side-view panoramic visible light image of the same scene area. The scene area in the side-view panoramic visible light image corresponds to the area within the frame of the DEM side view.

[0117] Specifically, the DEM image includes elevation and latitude / longitude information for each point in the scene area to be detected. The visible light camera that acquired the side-view visible light images of each sample also has elevation and latitude / longitude information. Based on this, the points in the DEM image are projected using the visible light camera as the origin. For example, the projection can be based on any projection algorithm. The position information of each point after projection is supplemented according to the elevation and latitude / longitude information, and the DEM image is converted into a DEM side view.

[0118] The location information determination method provided in this application uses multiple sample side-view visible light images of the scene area to be detected, acquired from different perspectives. The resulting side-view panoramic visible light image is then matched with a DEM side view converted from a DEM image. Compared to matching multiple single-view images separately with the DEM side view, this method reduces the number of matching points, simplifies the search for matching points, improves matching accuracy, and provides a more optimized global visualization effect. Converting the DEM image into a DEM side view also reduces the difficulty of obtaining matching points.

[0119] In one embodiment of this application, such as Figure 3-1 As shown, step S103 above performs dense mapping on the side-view panoramic visible light image and the DEM side view based on the scene point matching result to obtain a dense mapping relationship, including:

[0120] Step S301: Divide the side-view panoramic visible light image into multiple visible light regions according to the scene points, and divide the DEM side view into multiple DEM side view regions.

[0121] like Figure 3-2As shown, the side-view panoramic visible light image is divided into multiple triangular visible light regions, and the DEM side view is divided into multiple triangular DEM side view regions. Specifically, the triangular shape is only an example, and other shapes can also be used for region division, such as rectangles, irregular shapes, etc., as long as the same division is performed on the side-view panoramic visible light image and the DEM side view.

[0122] Step S302: Based on the scene point matching results, calculate the mapping transformation matrix between the visible light region and the DEM side view region that have a matching relationship, and obtain the dense mapping relationship.

[0123] Based on the scene point matching results between the side-view panoramic visible light image and the DEM side view obtained above, this result can represent the sparse mapping between the side-view panoramic visible light image and the DEM side view. Based on this, the mapping transformation matrix between the matched visible light region and the DEM side view region is calculated. For example, it can be an affine transformation, homography transformation, etc. Based on the mapping transformation matrix and the coordinates of the scene points in the DEM side view region, the coordinates of each point in the visible light region are calculated to establish a dense mapping relationship between the side-view panoramic visible light image and the DEM side view, such as... Figure 3-3 As shown.

[0124] The location information determination method provided in this application calculates the mapping transformation matrix between the visible light area and the side view area of ​​the DEM according to the division of the region, and establishes a dense mapping relationship between the regions one by one, so as to obtain a more complete and accurate dense mapping relationship between the side view panoramic visible light image and the side view of the DEM.

[0125] See Figure 4 This application also provides a flowchart of an event localization method, including:

[0126] Step S401: Detect preset type events in the first side-view visible light image of the scene area to be detected, and obtain the event detection results.

[0127] The preset event types are events that need to be located in the scene area to be detected, such as natural disaster events like fires, landslides, and flash floods. Preset event types can also be human activity events, such as the presence of humans in uninhabited areas or wildlife reserves. The first-side-view visible light image is an image captured by the image acquisition device used for monitoring in the scene area to be detected, such as real-time video screenshots from surveillance equipment or snapshots taken at fixed intervals by surveillance equipment.

[0128] The first-side view of the visible light image of the scene area to be detected is used to detect events of a preset type. The event detection results reflect whether the preset type of event that needs to be located exists in the scene area. The detection of preset type events in the visible light image can be achieved by deep learning algorithms, and this application does not specifically limit the deep learning algorithm used.

[0129] Step S402: If the event detection result indicates that a preset type of event exists in the first side-view visible light image, obtain the first viewpoint information when the camera captures the first side-view visible light image and the first region in the first side-view visible light image where the preset type of event occurs.

[0130] In the presence of a preset type of event, the viewpoint information, i.e., the first viewpoint information, is obtained when the camera captures the first side-view visible light image, and the first area in the scene region of the first side-view visible light image where the preset type of event occurs.

[0131] Step S403: Based on the first perspective information, determine the first mapping position of the first region in the side-view panoramic visible light image of the scene area to be detected.

[0132] Both the first side-view visible light image and the side-view panoramic visible light image are images of the scene area to be detected. Therefore, the coordinate transformation relationship between the first side-view visible light image and the side-view panoramic visible light image can be obtained based on their perspectives.

[0133] In one example, there are multiple scene regions to be detected, each corresponding to a side-view panoramic visible light image. By comparing the first side-view visible light image with the first viewing angle information, the corresponding side-view panoramic visible light image is determined, which is the image including the position information of the first region, and the first mapping position of the first region in the side-view panoramic visible light image is determined. For example, each scene region to be detected has its own index number in the corresponding side-view panoramic visible light image. The index number is used to distinguish each scene region to be detected. The corresponding index number is determined based on the first side-view visible light image and the first viewing angle information. The side-view panoramic visible light image is determined based on the index number, and the first mapping position of the first region in the side-view panoramic visible light image is determined.

[0134] Step S404: Based on the predetermined position information of each point in the side-view panoramic visible light image, the position information of the first mapped position is used as the location of a preset type event.

[0135] The position information of each point in the side-view panoramic visible light image is obtained by any of the position information determination methods described above.

[0136] Based on the position information of each point in the side-view panoramic visible light image determined in advance by the above position information determination method, the position information of the first mapping position is determined, thereby realizing the positioning of the preset type of event.

[0137] The event localization method provided in this application pre-calibrates the position information of each point in the side-view panoramic visible light image, separates the calibration process from the localization process, so that the calibration process can be completed offline, and the results are stored in the side-view panoramic visible light image. During the localization process, by using images from the same viewpoint and images of the area where the event occurred, rapid localization in large scenes can be achieved, increasing the localization distance, simplifying the localization difficulty, improving the efficiency of event localization, and providing a foundation for timely response to subsequent events.

[0138] In one embodiment of this application, the method further includes:

[0139] Acquire at least one auxiliary image of the first side-view visible light image, and acquire second view information of the auxiliary image, wherein the field of view of the auxiliary image includes the field of view corresponding to the first region, and the magnification of the auxiliary image is less than the magnification of the first side-view visible light image.

[0140] The auxiliary image refers to an image that helps confirm the first region corresponding to the side-view panoramic visible light image and the corresponding first mapped position. It has the same field of view as the first side-view visible light image, including the scene of the first region. The auxiliary image has a lower magnification than the first side-view visible light image; that is, the auxiliary image contains more scene areas than the first side-view visible light image, making it easier to identify the first region. For example, it can be acquired by other devices near the device acquiring the first side-view visible light image, or by a device capable of monitoring a portion of the area where the device acquiring the first side-view visible light image is located.

[0141] In one example, the auxiliary image can be an image with a different magnification at the same viewpoint as the first side-view visible light image, or it can be an image with the same field of view at a different viewpoint as the first side-view visible light image, and with the same viewpoint as a certain image region in the side-view panoramic visible light image where the first side-view visible light image is located.

[0142] Acquire one or more auxiliary images of the first side-view visible light image, and acquire second-view information of the auxiliary images.

[0143] Step S403 above, based on the first viewpoint information, determines the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected, including:

[0144] Based on the first perspective information and the second perspective information, determine the second mapping position of the first region in the auxiliary image;

[0145] Based on the second perspective information and the second mapping position, the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected is determined.

[0146] The event localization method provided in this application, based on a first side-view visible light image including the same field of view and its corresponding auxiliary image, determines the first mapping position of the first region in the side-view panoramic visible light image of the scene area to be detected according to multiple viewpoint information, which can achieve the determination of event location more quickly and accurately.

[0147] In one embodiment of this application, the first perspective information is first PTZ information;

[0148] The step of acquiring at least one auxiliary image of the first side-view visible light image and acquiring second-view information of the auxiliary image includes:

[0149] N auxiliary images of the first side-view visible light image are obtained, and the PTZ information of the auxiliary images is obtained, wherein N is an integer greater than 1, the P and T information of the first side-view visible light image is the same as that of the first N-1 auxiliary images, the magnification of the N-1th auxiliary image is the same as that of the Nth auxiliary image, the magnification of the 1st auxiliary image is greater than that of the first side-view visible light image, the magnification of the (i+1)th auxiliary image is greater than that of the 1st auxiliary image, i∈[1, N-1], and i is an integer.

[0150] PTZ information refers to the left / right / up / down movement of the acquisition device, lens magnification, and zoom control information, which are the perspective information of the image.

[0151] For example, when the first viewpoint information is P=0, T=5, Z=50, and the image with viewpoint information P=0, T=0, Z=1 has a common field of view (common view) with the first side-view visible light image, then the images with viewpoint information of viewpoint 1 (P=0, T=5, Z=25), viewpoint 2 (P=0, T=5, Z=12), viewpoint 3 (P=0, T=5, Z=6), viewpoint 4 (P=0, T=5, Z=3), viewpoint 5 (P=0, T=5, Z=2), viewpoint 6 (P=0, T=5, Z=1), and viewpoint 7 (P=0, T=0, Z=1) can all be auxiliary images, and all of the above viewpoints can be second viewpoint information.

[0152] Determining the second mapping position of the first region in the auxiliary image based on the first viewpoint information and the second viewpoint information includes:

[0153] Based on the first PTZ information and the PTZ information of the first auxiliary image, determine the mapping position of the first region in the first auxiliary image;

[0154] Based on the PTZ information of the i-th and (i+1)-th auxiliary images, the mapping position of the first region in the (i+1)-th auxiliary image is determined, wherein the second mapping position is the mapping position of the first region in the N-th auxiliary image.

[0155] The event localization method provided in this application determines a first side-view visible light image and its corresponding auxiliary image, which include the same field of view, based on the rules of view information. It determines the first mapping position of the first region in the side-view panoramic visible light image of the scene area to be detected based on multiple view information, which can achieve the determination of event location more quickly and accurately.

[0156] In one embodiment of this application, the flowchart of the above-described location information determination method and event location method is shown below. Figure 5 As shown, the location information determination method is the calibration stage, and the event localization method is the localization stage. In the calibration stage, 1-n captured images of the scene area to be detected are used as sample side-view visible light images, which are then stitched together to obtain a side-view panoramic visible light image. The DEM map is converted into a DEM side view based on camera point information. Scene point matching is performed on the side-view panoramic visible light image and the DEM side view to find matching point pairs and obtain a sequence of matching point pairs (scene point matching results), thus achieving sparse mapping. Based on the scene point matching results, dense mapping is performed on the side-view panoramic visible light image and the DEM side view to obtain a dense mapping relationship between the side-view panoramic visible light image and the DEM side view. Based on the dense mapping relationship and the location information of each point in the DEM side view, the location information of each point in the side-view panoramic visible light image is determined, and the location information is saved offline.

[0157] During the localization phase, the preset event type is a fire event, and the location of the preset event type is the fire point. The captured fire point image is used as the first side-view visible light image, and the fire point coordinates in it are the location of the first area where the fire event occurred. Through the first side-view visible light image and the auxiliary captured image (auxiliary image), the corresponding side-view panoramic visible light image and the fire point coordinates in it are determined, which is the second mapping position. The fire point's location information is found and determined by searching the location information of each point in the offline-saved side-view panoramic visible light image, thus realizing the localization of the fire point.

[0158] See Figure 6 This application also provides a schematic diagram of the structure of a location information determination device, including:

[0159] The image acquisition module 601 is used to acquire a side-view panoramic visible light image of the scene area to be detected, and a DEM side view, wherein the DEM side view includes the position information of each point in the scene area to be detected;

[0160] Scene point matching module 602 is used to match the same scene points in the side-view panoramic visible light image and the DEM side view to obtain scene point matching results;

[0161] The dense mapping module 603 is used to perform dense mapping on the side-view panoramic visible light image and the DEM side view based on the scene point matching result to obtain a dense mapping relationship;

[0162] The location information determination module 604 is used to determine the location information of each point in the side-view panoramic visible light image based on the location information of each point in the DEM side view and the dense mapping relationship.

[0163] The location information determination device provided in this application embodiment has high positioning accuracy because it performs dense mapping between the side-view panoramic visible light image and the DEM side view. Even in a large scene area at a long distance, if the number of matching points under dense mapping meets a certain requirement, it can still have high positioning accuracy. Based on the positioning information obtained in this way, the accuracy of positioning information in large scenes is improved.

[0164] In one embodiment of this application, the image acquisition module 601 is specifically used for:

[0165] Acquire sample side-view visible light images of the scene region to be detected from multiple different viewpoints using a visible light camera, where adjacent viewpoints have overlapping fields of view;

[0166] Based on the acquisition angle of the side-view visible light images of each sample, the side-view visible light images of each sample are stitched together in a panoramic view to obtain a side-view panoramic visible light image.

[0167] Obtain the DEM image of the scene area to be detected, wherein the DEM image includes the elevation information and latitude and longitude information of each point in the scene area to be detected;

[0168] A side view of the DEM is generated based on the elevation and latitude / longitude information of each point in the DEM image and the elevation and latitude / longitude information of the visible light camera.

[0169] The location information determination device provided in this application uses multiple sample side-view visible light images of the scene area to be detected, acquired from different perspectives. The resulting side-view panoramic visible light image is then matched with a DEM side view converted from a DEM image. Compared to matching multiple single-view images separately with the DEM side view, this method reduces the number of matching points, simplifies the difficulty of finding matching points, improves matching accuracy, and provides a more optimized global visualization effect. Converting the DEM image into a DEM side view also reduces the difficulty of obtaining matching points.

[0170] In one embodiment of this application, the dense mapping module 603 is specifically used for:

[0171] The side-view panoramic visible light image is divided into multiple visible light regions according to the scene points, and the DEM side view is divided into multiple DEM side view regions.

[0172] Based on the scene point matching results, the mapping transformation matrix between the visible light region and the side view region of the DEM with matching relationship is calculated to obtain the dense mapping relationship.

[0173] The location information determination device provided in this application calculates the mapping transformation matrix between the visible light area and the side view area of ​​the DEM according to the divided area, and establishes a dense mapping relationship between the areas one by one, so as to obtain a more complete and accurate dense mapping relationship between the side view panoramic visible light image and the side view of the DEM.

[0174] See Figure 7 This application also provides a schematic diagram of the structure of an event location device, including:

[0175] The event detection module 701 is used to detect preset types of events in the first side-view visible light image of the scene area to be detected, and obtain the event detection results;

[0176] The first region acquisition 702 is used to acquire, when the event detection result indicates that there is a preset type of event in the first side-view visible light image, the first view angle information when the camera captures the first side-view visible light image and the first region in the first side-view visible light image where the preset type of event occurs.

[0177] The first mapping position determination module 703 is used to determine the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected based on the first view information.

[0178] The event localization module 704 is used to locate the first mapped position as a preset type event based on the predetermined position information of each point in the side-view panoramic visible light image, wherein the position information of each point in the side-view panoramic visible light image is obtained by any of the aforementioned position information determination devices.

[0179] The event localization device provided in this application pre-calibrates the position information of each point in the side-view panoramic visible light image, separates the calibration process from the localization process, so that the calibration process can be completed offline, and the result is stored in the side-view panoramic visible light image. During the localization process, by using images from the same viewpoint and images of the area where the event occurred, rapid localization in large scenes can be achieved, increasing the localization distance, simplifying the localization difficulty, improving the efficiency of event localization, and providing a foundation for timely response to subsequent events.

[0180] In one embodiment of this application, the apparatus further includes:

[0181] The image information acquisition module is used to acquire at least one auxiliary image of the first side-view visible light image, and to acquire second view information of the auxiliary image, wherein the acquisition field of view of the auxiliary image includes the field of view corresponding to the first region, and the magnification of the auxiliary image is less than the magnification of the first side-view visible light image.

[0182] The first mapping position determination module 703 includes:

[0183] The second mapping position determination submodule is used to determine the second mapping position of the first region in the auxiliary image based on the first view information and the second view information;

[0184] The first mapping position determination submodule is used to determine the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected, based on the second view information and the second mapping position.

[0185] The event localization device provided in this application, based on a first side-view visible light image including the same field of view and its corresponding auxiliary image, determines the first mapping position of the first region in the side-view panoramic visible light image of the scene area to be detected according to multiple viewpoint information, which can more quickly and accurately determine the event location.

[0186] In one embodiment of this application, the first perspective information is first PTZ information;

[0187] The image information acquisition module is specifically used for:

[0188] N auxiliary images of the first side-view visible light image are obtained, and the PTZ information of the auxiliary images is obtained, wherein N is an integer greater than 1, the P and T information of the first side-view visible light image is the same as that of the first N-1 auxiliary images, the magnification of the N-1th auxiliary image is the same as that of the Nth auxiliary image, the magnification of the 1st auxiliary image is greater than that of the first side-view visible light image, the magnification of the (i+1)th auxiliary image is greater than that of the 1st auxiliary image, i∈[1, N-1], and i is an integer;

[0189] The second mapping position determination submodule is specifically used for:

[0190] Based on the first PTZ information and the PTZ information of the first auxiliary image, determine the mapping position of the first region in the first auxiliary image;

[0191] Based on the PTZ information of the i-th and (i+1)-th auxiliary images, the mapping position of the first region in the (i+1)-th auxiliary image is determined, wherein the second mapping position is the mapping position of the first region in the N-th auxiliary image.

[0192] The event localization device provided in this application determines a first side-view visible light image and its corresponding auxiliary image, which include the same field of view, based on the rules of viewpoint information. It also determines the first mapping position of a first region in the side-view panoramic visible light image of the scene area to be detected based on multiple viewpoint information, enabling faster and more accurate determination of event location.

[0193] This application also provides an electronic device, such as... Figure 8 As shown, it includes:

[0194] Memory 801 is used to store computer programs;

[0195] When the processor 802 executes the program stored in the memory 801, it implements any of the above-described location information determination methods and event localization methods.

[0196] Furthermore, the aforementioned electronic device may also include a communication bus and / or a communication interface, with the processor 802, the communication interface, and the memory 801 communicating with each other via the communication bus.

[0197] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0198] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0199] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0200] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0201] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the above-described location information determination methods and event location methods.

[0202] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the location information determination method and event location method in the above embodiments.

[0203] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a solid-state drive (SSD), etc.

[0204] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0205] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the embodiments for apparatus, electronic devices, and storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0206] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application are included within the scope of protection of this application.

Claims

1. An event localization method, characterized in that, include: The first side-view visible light image of the scene area to be detected is used to detect events of a preset type, and the event detection results are obtained. If the event detection result indicates that a preset type of event exists in the first side-view visible light image, the first viewpoint information when the camera captures the first side-view visible light image and the first region in the first side-view visible light image where the preset type of event occurs are obtained; Acquire at least one auxiliary image of the first side-view visible light image, and acquire second view information of the auxiliary image, wherein the acquisition field of view of the auxiliary image includes the field of view corresponding to the first region, and the magnification of the auxiliary image is less than the magnification of the first side-view visible light image; Based on the first perspective information and the second perspective information, determine the second mapping position of the first region in the auxiliary image; Based on the second perspective information and the second mapping position, determine the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected; Based on the predetermined position information of each point in the side-view panoramic visible light image, the position information of the first mapped position is used as the location of a preset type event; The process of pre-determining the positional information of each point in the side-view panoramic visible light image includes: A side-view panoramic visible light image of the scene area to be detected and a DEM side view are acquired. The DEM side view is generated based on the elevation and latitude / longitude information of each point in the DEM image of the scene area to be detected, and the elevation and latitude / longitude information of the visible light camera. The DEM image is a bird's-eye view, and the DEM side view includes the position information of each point in the scene area to be detected. Matching identical scene points in the side-view panoramic visible light image and the DEM side view yields scene point matching results. Based on the scene point matching results, a dense mapping is performed on the side-view panoramic visible light image and the DEM side view to obtain a dense mapping relationship. Based on the position information of each point in the DEM side view and the dense mapping relationship, the position information of each point in the side-view panoramic visible light image is determined.

2. The method according to claim 1, characterized in that, The first perspective information is the first PTZ information; The step of acquiring at least one auxiliary image of the first side-view visible light image and acquiring second-view information of the auxiliary image includes: N auxiliary images of the first side-view visible light image are obtained, and the PTZ information of the auxiliary images is obtained, wherein N is an integer greater than 1, the P and T information of the first side-view visible light image is the same as that of the first N-1 auxiliary images, the magnification of the N-1th auxiliary image is the same as that of the Nth auxiliary image, the magnification of the 1st auxiliary image is greater than that of the first side-view visible light image, the magnification of the (i+1)th auxiliary image is greater than that of the 1st auxiliary image, i∈[1, N-1], and i is an integer; Determining the second mapping position of the first region in the auxiliary image based on the first viewpoint information and the second viewpoint information includes: Based on the first PTZ information and the PTZ information of the first auxiliary image, determine the mapping position of the first region in the first auxiliary image; Based on the PTZ information of the i-th and (i+1)-th auxiliary images, the mapping position of the first region in the (i+1)-th auxiliary image is determined, wherein the second mapping position is the mapping position of the first region in the N-th auxiliary image.

3. The method according to claim 1, characterized in that, The acquisition of the side-view panoramic visible light image of the scene region to be detected includes: Acquire sample side-view visible light images of the scene region to be detected from multiple different viewpoints using a visible light camera, where adjacent viewpoints have overlapping fields of view; Based on the acquisition angle of the side-view visible light images of each sample, the side-view visible light images of each sample are stitched together in a panoramic view to obtain a side-view panoramic visible light image.

4. The method according to claim 1, characterized in that, The step of performing dense mapping on the side-view panoramic visible light image and the DEM side view based on the scene point matching result to obtain a dense mapping relationship includes: The side-view panoramic visible light image is divided into multiple visible light regions according to the scene points, and the DEM side view is divided into multiple DEM side view regions. Based on the scene point matching results, the mapping transformation matrix between the visible light region and the side view region of the DEM with matching relationship is calculated to obtain the dense mapping relationship.

5. An event location device, characterized in that, include: The event detection module is used to detect preset types of events in the first side view visible light image of the scene area to be detected, and obtain the event detection results; The first region acquisition is used to acquire, when the event detection result indicates that there is a preset type of event in the first side-view visible light image, the first view angle information when the camera captures the first side-view visible light image and the first region in the first side-view visible light image where the preset type of event occurs; The image information acquisition module is used to acquire at least one auxiliary image of the first side-view visible light image, and to acquire second view information of the auxiliary image, wherein the acquisition field of view of the auxiliary image includes the field of view corresponding to the first region, and the magnification of the auxiliary image is less than the magnification of the first side-view visible light image. The first mapping location determination module includes: The second mapping position determination submodule is used to determine the second mapping position of the first region in the auxiliary image based on the first view information and the second view information; The first mapping position determination submodule is used to determine the first mapping position of the first region in the side-view panoramic visible light image of the scene region to be detected, based on the second view information and the second mapping position. The event localization module is used to locate the first mapped position as a preset type event based on the predetermined position information of each point in the side-view panoramic visible light image. The image acquisition module is used to acquire a side-view panoramic visible light image of the scene area to be detected, and a DEM side view. The DEM side view is generated based on the elevation and latitude and longitude information of each point in the DEM image of the scene area to be detected, and the elevation and latitude and longitude information of the visible light camera. The DEM image is a bird's-eye view, and the DEM side view includes the position information of each point in the scene area to be detected. The scene point matching module is used to match the same scene points in the side-view panoramic visible light image and the DEM side view to obtain scene point matching results. The dense mapping module is used to perform dense mapping on the side-view panoramic visible light image and the DEM side view based on the scene point matching results, so as to obtain a dense mapping relationship; The location information determination module is used to determine the location information of each point in the side-view panoramic visible light image based on the location information of each point in the DEM side view and the dense mapping relationship.

6. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method of any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-4.

Images