Abnormality positioning method, apparatus, device, and readable storage medium

Abstract

Embodiments of the present application provide an abnormality positioning method, device and equipment and readable storage medium, the method comprises: collecting the area video of a target area; image processing the area video to obtain the abnormal video frame corresponding to the abnormal point in the target area; extracting features from the abnormal video frame to obtain the abnormal feature point corresponding to the abnormal video frame; mapping the abnormal feature point to the area model of the target area to position the abnormal point in the area model.

Description

Technical Field

[0001] This application relates to the field of target positioning technology, and includes, but is not limited to, an anomaly positioning method, apparatus, device, and readable storage medium. Background Technology

[0002] Currently, many high-definition cameras are deployed in factories, such as power line inspection systems, to detect defects in various equipment, such as rust and oil leaks. These technologies are all based on the captured video frames. However, to pinpoint the exact location of a defect using camera video frames, it's necessary to first determine the location of the camera that reported the anomaly, then check the location of the equipment monitored by the camera, and finally investigate within the monitored equipment to locate the anomaly. This anomaly localization process is lengthy and prone to errors when performing anomaly searches in large factories with similar equipment. Summary of the Invention

[0003] Based on the problems existing in related technologies, embodiments of this application provide an anomaly location method, apparatus, device, and readable storage medium.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides an anomaly localization method, the method comprising:

[0006] Collect regional video of the target area;

[0007] Image processing is performed on the video of the region to obtain abnormal video frames corresponding to the abnormal points in the target region;

[0008] Feature extraction is performed on the abnormal video frame to obtain the abnormal feature points corresponding to the abnormal video frame;

[0009] The abnormal feature points are mapped onto the region model of the target region to locate the abnormal points in the region model.

[0010] In some embodiments, the step of performing image processing on the regional video to obtain abnormal video frames corresponding to abnormal points in the target region includes:

[0011] The region video is feature-labeled to obtain the reference video frame corresponding to the region video;

[0012] Each video frame of the video in the region and the reference video frame are processed to obtain the target grayscale image and the reference grayscale image respectively.

[0013] The target grayscale image and the reference grayscale image are subjected to similarity processing, and the target grayscale image with a similarity lower than a preset threshold is determined as an abnormal grayscale image;

[0014] The video frame corresponding to the abnormal grayscale image is identified as the abnormal video frame.

[0015] In some embodiments, the video region where the abnormal video frame is located corresponds to a reference video frame;

[0016] The step of extracting features from the abnormal video frames to obtain the abnormal feature points corresponding to the abnormal video frames includes:

[0017] The regions in the abnormal video frames that are different from the reference video frames are identified as abnormal regions.

[0018] Edge feature extraction and region shape feature extraction are performed on the abnormal region to obtain the region feature points of the abnormal region, and the region feature points are determined as the abnormal feature points.

[0019] In some embodiments, the target region has multiple regional feature points and multiple video acquisition points at different locations, each video acquisition point corresponding to an acquisition coordinate, and regional video corresponding to the target region is acquired at each of the video acquisition points; the method further includes:

[0020] In multiple regional videos, at least two regional videos corresponding to each regional feature point are determined, and at least two regional video frames corresponding to the at least two regional videos are determined.

[0021] Determine the feature coordinates of each region feature point in each of the at least two region video frames;

[0022] Calculate the three-dimensional coordinates of each feature point in the region based on the acquired coordinates and the feature coordinates;

[0023] The region model is obtained by surface triangulation based on the three-dimensional coordinates corresponding to each feature point in the region.

[0024] In some embodiments, the method further includes:

[0025] Obtain the vertex coordinates of each surface triangle on the surface of the region model after surface triangulation, the region video corresponding to each surface triangle, and the acquisition coordinates of the video acquisition point corresponding to the region video;

[0026] Based on the acquired coordinates and the vertex coordinates, determine the texture coordinates corresponding to the vertex coordinates in the regional video;

[0027] Based on the texture coordinates, determine the texture image corresponding to the texture coordinates in the regional video;

[0028] The surface triangle is rendered based on the texture image to obtain a real-world region model; wherein the real-world region model is a model capable of representing the real-world image of the target region.

[0029] In some embodiments, the video area where the abnormal video frame is located has a corresponding video acquisition point, and the video acquisition point has acquisition coordinates;

[0030] The step of mapping the abnormal feature points onto a region model of the target region to locate the abnormal points in the region model includes:

[0031] Determine the first abnormal coordinates of the abnormal feature point in the abnormal video frame;

[0032] Based on the collected coordinates and the first abnormal coordinates, the abnormal feature points are mapped onto the regional model of the target area to locate the abnormal points in the regional model.

[0033] In some embodiments, the region model surface is composed of multiple surface triangles; the method further includes:

[0034] After mapping the abnormal feature points to the region model, at least one surface triangle containing the abnormal feature points and the vertex coordinates of the at least one surface triangle are determined.

[0035] Based on the acquisition coordinates and the vertex coordinates, determine the abnormal image corresponding to the at least one surface triangle in the abnormal video frame;

[0036] The at least one surface triangle is rendered based on the anomalous image to display the anomalous image on the surface of the region model.

[0037] This application provides an anomaly location device, the device comprising:

[0038] The acquisition module is used to acquire regional video of the target area;

[0039] The image processing module is used to perform image processing on the video of the region to obtain abnormal video frames corresponding to abnormal points in the target region;

[0040] The feature extraction module is used to extract features from the abnormal video frame to obtain the abnormal feature points corresponding to the abnormal video frame.

[0041] An anomaly localization module is used to map the abnormal feature points onto a region model of the target region, so as to locate the abnormal points in the region model.

[0042] The anomaly location device provided in this application includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements the anomaly location method described in this application.

[0043] The computer-readable storage medium provided in this application embodiment stores executable instructions thereon, which are used to cause a processor to execute the executable instructions to implement the anomaly location method provided in this application embodiment.

[0044] The anomaly localization method, apparatus, device, and readable storage medium provided in this application acquire regional video of a target area to obtain anomaly video frames corresponding to anomaly points, and map the anomaly feature points corresponding to the anomaly video frames onto a regional model of the target area to perform anomaly localization on the regional model. This application embodiment intuitively locates the anomaly points in the video frames onto the regional model of the target area, quickly determining the defect location of the target area. In scenarios with similar equipment, it can quickly find the anomaly location, improve the efficiency of anomaly localization, and reduce labor costs. Attached Figure Description

[0045] Figure 1 This is an optional flowchart illustrating the anomaly localization method provided in an embodiment of this application;

[0046] Figure 2 This is a schematic diagram illustrating an application scenario of the anomaly localization method provided in the embodiments of this application;

[0047] Figure 3 This is an optional flowchart illustrating the anomaly localization method provided in an embodiment of this application;

[0048] Figure 4 This is an optional flowchart illustrating the anomaly localization method provided in an embodiment of this application;

[0049] Figure 5 This is an optional flowchart illustrating the anomaly localization method provided in an embodiment of this application;

[0050] Figure 6 This is a schematic diagram of the composition structure of the anomaly location device provided in the embodiments of this application;

[0051] Figure 7 This is a schematic diagram of the composition structure of the anomaly location device provided in the embodiments of this application. Detailed Implementation

[0052] To more clearly illustrate the purpose, technical solutions, and advantages of the embodiments of this application, the embodiments of this application will be described in detail below with reference to the accompanying drawings. It should be understood that the following description of the embodiments is intended to explain and illustrate the overall concept of the embodiments of this application, and should not be construed as a limitation of the embodiments of this application. In the specification and drawings, the same or similar reference numerals refer to the same or similar parts or components. For clarity, the drawings are not necessarily drawn to scale, and some well-known parts and structures may be omitted from the drawings.

[0053] In some embodiments, unless otherwise defined, the technical or scientific terms used in the embodiments of this application shall have the ordinary meaning understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. The word "a" or "an" does not exclude multiple components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," "right," "top," or "bottom" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. When an element such as a layer, film, region, or substrate is referred to as being "above" or "below" another element, the element may be "directly" located "above" or "below" the other element, or there may be intermediate elements present.

[0054] When using cameras to detect equipment anomalies, related technologies rely on manual querying of video frames. However, the images viewed through monitoring systems are typically two-dimensional, making it difficult to establish an intuitive connection with the real three-dimensional world being monitored. For example, when the area to be monitored is a power patrol system, there are many similar scenes and equipment, resulting in many similar areas. It is easy to make mistakes in positioning using two-dimensional monitoring images. After detecting an anomaly, it is difficult to quickly locate the specific spatial location of the anomaly. Searching manually time and time again is not only inefficient but also wastes a lot of manpower.

[0055] Based on the problems existing in related technologies, this application provides an anomaly localization method. By acquiring regional video of the target area, the method obtains the abnormal video frames corresponding to the abnormal points, and maps the abnormal feature points corresponding to the abnormal video frames onto the regional model of the target area to perform anomaly localization on the regional model. This method intuitively locates the abnormal points in the video frames onto the regional model of the target area, quickly determines the defect location of the target area, and can quickly find the abnormal location in scenarios with similar devices, thus improving the efficiency of anomaly localization.

[0056] The following describes exemplary applications of the anomaly location device provided in the embodiments of this application. The anomaly location device provided in the embodiments of this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, and mobile devices, or it can be implemented as a server. The following will describe exemplary applications when the anomaly location device is implemented as a server.

[0057] See Figure 1 , Figure 1 This is an optional flowchart illustrating the anomaly localization method provided in this application embodiment, which will be combined with... Figure 1 The steps shown are explained.

[0058] Step S101: Collect regional video of the target area.

[0059] Here, the target area can be a space that needs to be located for anomalies, such as a power plant or any operating factory. One or more cameras can be set up in the target area to collect monitoring videos of the target area, i.e., area videos of the target area. There can be multiple area videos, and every location in the target area can be monitored through multiple area videos.

[0060] In some embodiments, a single mobile acquisition device can be used to acquire videos of multiple different areas, or cameras can be set up at multiple different acquisition points to acquire videos of multiple different areas.

[0061] Step S102: Perform image processing on the video of the region to obtain abnormal video frames corresponding to the abnormal points in the target region.

[0062] In some embodiments, an anomaly point is the location of a defect in the target area, such as the location of an oil leak on a device. An anomaly video frame can be a video frame recorded from the area where the device has an oil leak. In this embodiment, multiple video capture devices can record the device experiencing an anomaly in the target area; therefore, multiple anomaly video frames from different perspectives can be obtained.

[0063] In some embodiments, abnormal video frames appearing in a region of video can be obtained using video target recognition methods such as frame difference or grayscale methods. Frame difference methods obtain abnormal video frames by performing a difference operation on two adjacent frames in the region of video, while grayscale methods obtain abnormal video frames by comparing a reference video frame with video frames in the region of video. It should be noted that any abnormal video frame detection method can be used to extract abnormal video frames in the embodiments of this application.

[0064] Step S103: Extract features from the abnormal video frame to obtain the abnormal feature points corresponding to the abnormal video frame.

[0065] In some embodiments, an abnormal region can be identified in an abnormal video frame, and features can be extracted from the abnormal region to obtain feature information such as the edge, shape, or texture of the abnormal region. Based on this feature information, abnormal feature points that can characterize the abnormal information can be determined. For example, after feature extraction of the abnormal region, the shape of the abnormal region is found to be hexagonal. By extracting the edge features of the hexagon, each corner and center point of the hexagon can be used as an abnormal feature point of the abnormal region.

[0066] In some embodiments, abnormal regions can be marked in abnormal video frames in the manner of bounding boxes using a defect detection algorithm, or abnormal regions can be marked on the abnormal video frame based on pixels, and then abnormal feature points can be extracted.

[0067] Step S104: Map the abnormal feature points onto the region model of the target region to locate the abnormal points in the region model.

[0068] In this embodiment, the target area's region model refers to a 3D real-world model of the target area reconstructed using 3D reconstruction technology. This model can reflect all the features of the target area. By determining the position of the abnormal video frame on the region model, and then mapping the abnormal feature points onto the region model, the abnormal feature points are displayed in the region model.

[0069] In some embodiments, the target region may have multiple regional feature points, which are points used to characterize regional features and perform model reconstruction. For example, when the target region is a factory, there are intersecting roads in the factory, and different instruments and equipment on each road. Each road has multiple regional feature points, which can characterize the width, length and intersection information between different roads, etc. Each surface of the instrument and equipment has multiple regional feature points, and each regional feature point constitutes the instrument and equipment. The regional feature points on the instrument and equipment, the regional feature points on the roads, and the relative positions between different regional feature points together characterize the entire target region.

[0070] In some embodiments, the target area corresponds to multiple video capture points at different locations. Each video capture point may have a video capture device. For example, cameras are set at multiple locations in the target area, and the cameras cover every location in the target area. Video of the target area is captured at each video capture point. The images in the regional videos captured by adjacent video capture points may overlap. For example, the same device may appear in at least two regional videos.

[0071] In this embodiment of the application, reconstructing the region model of the target area using three-dimensional reconstruction technology can be achieved through the following steps:

[0072] Step S1: Determine at least two regional videos corresponding to each regional feature point and at least two regional video frames corresponding to the at least two regional videos in multiple regional videos.

[0073] In this embodiment of the application, the reconstruction of the target region can be achieved by using the Structure from Motion (SFM) algorithm.

[0074] In some embodiments, each feature point of the target area can be acquired by multiple video acquisition devices. In the multiple area videos corresponding to each feature point, multiple area video frames corresponding to each feature point are determined. For example, a device has three area videos with different perspectives. The device has multiple area feature points. Through image recognition, area video frames with all area feature points of the device can be determined in the three area videos respectively. Each area video frame has a different acquisition perspective of the device.

[0075] Step S2: Determine the feature coordinates of each region feature point in each region video frame of the at least two region video frames.

[0076] In this embodiment, a spatial coordinate axis can be set for the target area. Based on the relative position of each video capture point within the target area, a capture coordinate is set for each video capture point on the spatial coordinate axis. The regional feature points in the video captured by each video capture point also have corresponding feature coordinates relative to this spatial coordinate axis. Specifically, the video frames in the video captured by each video capture point have a relative position to that video capture point. The coordinates of the video frames on the spatial coordinate axis can be calculated using triangulation. Based on the relative position and the capture coordinates of each video capture point, the feature coordinates of the regional feature points in the video frames captured by each video capture point can be obtained.

[0077] In some embodiments, feature points in the same area can be captured by video capture devices from at least two different perspectives. Therefore, the feature coordinates of the same feature points in the area video corresponding to different video capture points are different, that is, the same feature points in the same area have at least two feature coordinates.

[0078] Step S3: Calculate the three-dimensional coordinates of each feature point in the region based on the acquired coordinates and the feature coordinates.

[0079] In this embodiment of the application, the three-dimensional coordinates of each feature point in a region can be calculated by triangulation. For example, based on multiple acquisition coordinates and multiple feature coordinates corresponding to the feature points in the region, rays emanating from different video acquisition points can be established. The rays pass through the feature coordinates in the video frame of the region corresponding to the video acquisition point. The intersection of multiple rays of the same feature point in the region is the spatial coordinate point corresponding to the feature point in the region, and the coordinates of the spatial coordinate point are the three-dimensional coordinates of the feature point in the region.

[0080] In some embodiments, calculation errors may occur, and the rays may not converge to a single point. In such cases, the spatial coordinate point closest to all rays can be determined as the three-dimensional spatial coordinate point corresponding to the feature point of the region. The coordinates of this spatial coordinate point are the three-dimensional coordinates of the feature point of the region.

[0081] In this embodiment of the application, triangulation is performed sequentially on each feature point of the target area to obtain the three-dimensional spatial coordinates and corresponding three-dimensional coordinates of each feature point.

[0082] Step S4: Based on the three-dimensional coordinates corresponding to each feature point in the region, obtain the region model through surface triangulation.

[0083] In the embodiments of this application, surface triangulation refers to connecting three adjacent spatial coordinate points to form a triangle. After performing surface triangulation on all spatial coordinate points obtained by the triangulation method, a region model composed of multiple interconnected triangles is obtained, and the surface of the region model is composed of triangles.

[0084] In this embodiment of the application, after obtaining the region model after surface triangulation, the vertex coordinates of each surface triangle on the surface of the region model, the region video corresponding to each surface triangle, and the acquisition coordinates of the video acquisition points corresponding to the region video are obtained. Based on the acquisition coordinates and vertex coordinates, the texture coordinates corresponding to the vertex coordinates are determined in the region video. Based on the texture coordinates, the texture image corresponding to the texture coordinates is determined in the region video. The surface triangles are rendered based on the texture image to obtain the real-scene region model, wherein the real-scene region model is a model that can represent the real-scene image of the target region.

[0085] In this embodiment, based on each surface triangle and its vertex coordinates, the video capture point opposite each surface triangle and the area video captured by that capture point can be determined. A video frame is selected from the area video, and the vertex coordinates of the surface triangles on the area model are projected onto the video frame using triangulation to obtain three texture coordinates corresponding to the vertex coordinates of the surface triangles. Based on these texture coordinates, the corresponding texture image for the surface triangle is determined in the video frame, and the texture image is rendered onto the surface triangle of the area model, giving the triangle a texture image. Rendering each surface triangle on the surface of the area model yields a real-world area model, enabling the real-world area model to represent the real-world image of the target area.

[0086] In some embodiments, when rendering the surface of a region model, rendering can be performed in real time or at preset intervals, such as once every half hour.

[0087] This application embodiment acquires regional video of the target area to obtain abnormal video frames corresponding to abnormal points, and maps the abnormal feature points corresponding to the abnormal video frames onto the regional model of the target area to perform abnormal location on the regional model. This application embodiment intuitively locates the abnormal points in the video frames onto the regional model of the target area, and can quickly find the abnormal location in scenarios with similar devices, thus improving the efficiency of abnormal location.

[0088] See Figure 2 , Figure 2 This is a schematic diagram illustrating an application scenario of the anomaly localization method provided in this application embodiment. The anomaly localization system 20 provided in this application embodiment includes a terminal 100, a network 200, and a server 300. The server 300 obtains the regional video of the target area reported by the video acquisition point on the terminal 100 through the network 200, performs image processing on the regional video to obtain the abnormal video frames corresponding to the abnormal points in the target area and the abnormal feature points corresponding to the abnormal video frames. The abnormal feature points are mapped onto the regional model of the target area to obtain the abnormal location on the regional model. The abnormal location on the regional model is then sent to the terminal 100 through the network 200 to achieve anomaly localization of the abnormal points in the regional model. After the terminal 100 receives the abnormal location on the regional model, it can directly display the abnormal location on the current interface 100-1 to intuitively determine the location where the anomaly occurred in the target area.

[0089] Based on the foregoing embodiments, Figure 3 This is an optional flowchart illustrating the anomaly localization method provided in an embodiment of this application, such as... Figure 3 As shown, in some embodiments, step S102 can be implemented through the following steps:

[0090] Step S301: Perform feature calibration on the region video to obtain the reference video frame corresponding to the region video.

[0091] In some embodiments, the target area has multiple regional videos, each capturing a video image of one area within the target area. A reference video frame is determined by performing feature recognition on each video frame of each regional video. If a video frame contains all the features of the area corresponding to that regional video and has no abnormal features, that video frame is determined as the reference video frame for that regional video. Here, "no abnormalities" means that the equipment in the scene is operating normally, without any problems that should not occur, such as oil leaks.

[0092] Step S302: Perform image grayscale processing on each video frame of the region video and the reference video frame respectively to obtain the target grayscale image and the reference grayscale image.

[0093] In this embodiment of the application, image grayscale processing may involve simplifying the colors of each video frame of the reference video frame and the regional video and converting them into grayscale. That is, each pixel of each video frame of the reference video frame and the regional video retains only different degrees of grayscale, and each pixel has a corresponding grayscale value, thereby obtaining a reference grayscale image corresponding to the reference video frame and multiple target grayscale images corresponding to each video frame of the regional video.

[0094] Step S303: Perform similarity processing on the target grayscale image and the reference grayscale image, and determine the target grayscale image with a similarity lower than a preset threshold as an abnormal grayscale image.

[0095] In this embodiment, the similarity between a target grayscale image and a reference grayscale image can be determined using any image similarity calculation method. For example, the number of pixels in the target grayscale image whose grayscale values ​​differ from those of corresponding pixels in the reference grayscale image can be determined, and the ratio between the number of pixels with different grayscale values ​​and the total number of pixels in the reference image can be used to determine the similarity between the target grayscale image and the reference grayscale image. The preset threshold can be set based on the total number of pixels. When the total number of pixels is very large, such as 1980*1080, the preset threshold can be set lower, such as 90%; when the total number of pixels is not large, such as 640*480, the preset threshold can be set higher, such as 95%.

[0096] In this embodiment of the application, a target grayscale image whose similarity to a reference grayscale image is lower than a preset threshold among multiple target grayscale images is identified as an abnormal grayscale image.

[0097] Step S304: Determine the video frame corresponding to the abnormal grayscale image as the abnormal video frame.

[0098] This application embodiment uses image processing to identify abnormal video frames recorded in the target area of ​​the video, avoiding the need for manual frame-by-frame searching of abnormal video frames in the video, thus improving work efficiency and reducing labor costs.

[0099] In some embodiments, after identifying abnormal video frames, feature extraction is performed on the abnormal video frames to obtain abnormal regions within them. Based on the foregoing embodiments, Figure 4 This is an optional flowchart illustrating the anomaly localization method provided in an embodiment of this application, such as... Figure 4 As shown, in some embodiments, step S103 can be implemented through the following steps:

[0100] Step S401: Determine the regions in the abnormal video frames that are different from the reference video frames as abnormal regions.

[0101] Step S402: Extract edge features and shape features from the abnormal region to obtain the region feature points of the abnormal region, and determine the region feature points as the abnormal feature points.

[0102] In the embodiments of this application, any feasible image recognition method can be used to identify regions in an abnormal video frame that are different from a reference video frame, and these regions can be identified as abnormal regions.

[0103] In some embodiments, after identifying an abnormal region in an abnormal video frame, edge feature extraction and region shape feature extraction can be performed on the abnormal region to obtain feature information such as the edge, shape, or texture of the abnormal region, and abnormal feature points that can characterize the abnormal information can be determined based on these feature information. For example, after feature extraction of the abnormal region, the shape of the abnormal region is found to be hexagonal. By extracting the edge features of the hexagon, each corner and center point of the hexagon can be used as an abnormal feature point of the abnormal region.

[0104] In some embodiments, after identifying anomalous feature points, mapping these points onto a region model allows the precise location of the anomalous points to be determined on the region model. Based on the foregoing embodiments, Figure 5 This is an optional flowchart illustrating the anomaly localization method provided in an embodiment of this application, such as... Figure 5 As shown, in some embodiments, step S104 can be implemented through the following steps:

[0105] Step S501: Determine the first abnormal coordinates of the abnormal feature point in the abnormal video frame.

[0106] Step S502: Based on the collected coordinates and the first abnormal coordinates, map the abnormal feature points onto the region model of the target area to locate the abnormal points in the region model.

[0107] In this embodiment of the application, after forming the region model of the target region and the abnormal feature points in the video frame, the first abnormal coordinates of the abnormal feature points in the abnormal video frame are determined in the spatial coordinate axis, and the abnormal feature points are mapped onto the region model by the triangulation method, so as to display the abnormality of the target region on the region model.

[0108] In some embodiments, after mapping the abnormal feature points to the region model, the vertex coordinates of at least one surface triangle where the abnormal point is located and all surface triangles on the region model are determined. The abnormal image corresponding to at least one surface triangle is determined in the abnormal video frame where the abnormal feature point is located by triangulation. The abnormal image is then rendered onto the at least one surface triangle where the abnormal point is located, so that the surface of the region model can display the abnormal image.

[0109] In some embodiments, the region model can be a real-world region model. Here, the image of the surface triangle where the anomaly point is located can be replaced by an anomaly image so that the surface of the real-world region model can display the anomaly image.

[0110] After determining the specific location of the anomaly on the region model, this embodiment of the application renders the anomaly image in the video frame onto the model surface using triangulation, allowing technicians to intuitively see the anomaly in the target area, improving the efficiency of defect spatial localization and avoiding errors.

[0111] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0112] This application's embodiments are based on visual triangulation to reconstruct a 3D model of the actual scene (i.e., the actual scene area model) of the site (i.e., the target area). Simultaneously, a 3D digital map of the site is reconstructed using Simultaneous Localization and Mapping (SLAM) technology. This 3D digital map establishes a 3D digital space for the target monitoring area (i.e., the target area). Here, the 3D actual scene model is a 3D model (i.e., the area model) generated through triangulation (i.e., surface triangulation) and texture mapping. This 3D model can be rendered using a real-time 3D rendering engine to enhance the model's details and scene realism.

[0113] In some embodiments, triangulation can be implemented using a motion reconstruction structure algorithm. For example, feature points of the target region can be extracted first, then a set of scenes (which can be two images) can be selected, the camera pose can be estimated based on these two images, and the 3D coordinates of the feature points can be reconstructed. The 3D coordinates can be optimized, and finally, the above steps can be repeated for each of the remaining scenes to perform 3D reconstruction and obtain a spatial model.

[0114] In this embodiment of the application, the pose of the video frames in the surveillance video (i.e., the area video) of the target monitoring area in the digital map can be calculated by SLAM relocation technology to determine the pose of the camera (the acquisition device of the video acquisition point) in the target monitoring area in three-dimensional space, and a twin mirror of the camera can be established in the digital world (i.e., the acquisition coordinates of the video acquisition point are determined).

[0115] In this embodiment, a defect detection module can be used to process the monitoring video of the target monitoring area. When a defect is detected in the monitoring video, the defect is marked on the video frame by a defect detection algorithm in the form of a bounding box or by pixels. Since the pose of the camera in the three-dimensional digital space has been calculated, the marked defect area can be projected into the three-dimensional digital space by the back projection method, and the surface area of ​​the spatial model where the defect area is located can be calculated.

[0116] In some embodiments, the corresponding area on the model surface can be updated using the video frame where the defect is located, so that the model surface is updated to display the defect. Users can intuitively locate and observe the specific location and state of the defect in the 3D scene.

[0117] In some embodiments, the 3D reality model is typically a 3D model whose surface has been triangulated and textured. Textures enhance model details and scene realism. The triangulated model can be rendered using a real-time 3D rendering engine. The texture coordinates of the triangular facets in the video frame can be calculated based on the projection of the vertices of the triangular facets onto the video frame. The textures of the triangular facets are then updated based on the images corresponding to the texture coordinates. The 3D model is rendered and displayed using the new textures, allowing defective areas to be updated in real time and displayed in the 3D reality model.

[0118] In some embodiments, calculating the model surface area covered by the defect can first involve emitting a ray from the defect edge pixel in the video frame, originating from the camera position corresponding to the video frame. The intersection point of the ray with the model surface is calculated. Multiple rays corresponding to the defect edge pixels are then used to generate a defect-covered area on the model surface. This defect-covered surface area is then segmented for subsequent texture updates. Next, the texture coordinates of each triangle vertex in the defect-covered area are calculated within the video frame. After obtaining the texture coordinates, the corresponding texture image is rendered onto the defect-covered area of ​​the model surface.

[0119] This application proposes to establish a digital twin world of the camera monitoring area. By using 3D reconstruction technology, a 3D real-scene model of the monitoring area and a 3D digital map for monitoring equipment and anomaly location are reconstructed. Camera data is registered in the 3D space and the camera's pose in space is calculated. The monitoring area, camera, and camera monitoring screen are all registered in the same 3D digital space. In this way, the anomaly images detected in real time by the monitoring data can be mapped onto the 3D real-scene model. The location of the anomaly can be directly obtained and presented in the 3D real-scene model, improving the efficiency of defect spatial location and avoiding errors.

[0120] Based on the above anomaly localization methods Figure 6 This is a schematic diagram of the structural composition of the anomaly location device provided in the embodiments of this application, as shown below. Figure 6 As shown, the anomaly localization device 600 includes an acquisition module 601, an image processing module 602, a feature extraction module 603, and an anomaly localization module 604. The acquisition module 601 is used to acquire regional video of a target area; the image processing module 602 is used to perform image processing on the regional video to obtain abnormal video frames corresponding to anomalies in the target area; the feature extraction module 603 is used to extract features from the abnormal video frames to obtain abnormal feature points corresponding to the abnormal video frames; and the anomaly localization module 604 is used to map the abnormal feature points onto a regional model of the target area to locate the anomalies in the regional model.

[0121] In some embodiments, the image processing module 602 is further configured to perform feature calibration on the regional video to obtain a reference video frame corresponding to the regional video; perform image grayscale processing on each video frame of the regional video and the reference video frame respectively to obtain a target grayscale image and a reference grayscale image; perform similarity processing on the target grayscale image and the reference grayscale image, and determine the target grayscale image with a similarity lower than a preset threshold as an abnormal grayscale image; and determine the video frame corresponding to the abnormal grayscale image as the abnormal video frame.

[0122] In some embodiments, the region video where the abnormal video frame is located corresponds to a reference video frame; the feature extraction module 603 is further configured to determine the region in the abnormal video frame that is different from the reference video frame as an abnormal region; perform edge feature extraction and region shape feature extraction on the abnormal region to obtain the region feature points of the abnormal region, and determine the region feature points as the abnormal feature points.

[0123] In some embodiments, the target region has multiple regional feature points and multiple video acquisition points at different locations, each video acquisition point corresponding to an acquisition coordinate, and regional video corresponding to the target region is acquired at each of the video acquisition points; the anomaly localization device further includes a first determining module, used to determine at least two regional videos corresponding to each regional feature point and at least two regional video frames corresponding to the at least two regional videos in the multiple regional videos; a second determining module, used to determine the feature coordinates of each regional feature point in each of the at least two regional video frames; a calculation module, used to calculate the three-dimensional coordinates of each regional feature point according to the acquisition coordinates and the feature coordinates; and a triangulation processing module, used to obtain the region model by surface triangulation processing according to the three-dimensional coordinates corresponding to each regional feature point.

[0124] In some embodiments, the anomaly localization device further includes an acquisition module, configured to acquire the vertex coordinates of each surface triangle on the surface of the region model after surface triangulation, the region video corresponding to each surface triangle, and the acquisition coordinates of the video acquisition point corresponding to the region video; a third determination module, configured to determine the texture coordinates corresponding to the vertex coordinates in the region video based on the acquisition coordinates and the vertex coordinates; a fourth determination module, configured to determine the texture image corresponding to the texture coordinates in the region video based on the texture coordinates; and a rendering module, configured to render the surface triangles based on the texture image to obtain a real-world region model; wherein the real-world region model is a model capable of representing the real-world image of the target region.

[0125] In some embodiments, the video area where the abnormal video frame is located has a corresponding video acquisition point, and the video acquisition point has acquisition coordinates; the abnormal location module 604 is further configured to determine the first abnormal coordinates of the abnormal feature point in the abnormal video frame; and map the abnormal feature point onto the region model of the target region according to the acquisition coordinates and the first abnormal coordinates, so as to perform abnormal location of the abnormal point in the region model.

[0126] In some embodiments, the surface of the region model is composed of multiple surface triangles; the anomaly localization device further includes a fifth determining module, configured to determine at least one surface triangle containing the anomaly feature point and the vertex coordinates of the at least one surface triangle after mapping the anomaly feature point to the region model; a sixth determining module, configured to determine the anomaly image corresponding to the at least one surface triangle in the anomaly video frame based on the acquired coordinates and the vertex coordinates; and a rendering module, configured to render the at least one surface triangle based on the anomaly image to display the anomaly image on the surface of the region model.

[0127] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment; therefore, it will not be repeated. For technical details not disclosed in this apparatus embodiment, please refer to the description of the method embodiment of this application for understanding.

[0128] It should be noted that, in the embodiments of this application, if the above-mentioned anomaly location method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a terminal to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware and software combination.

[0129] This application provides an anomaly location device. Figure 7 This is a schematic diagram of the structural composition of the anomaly location device provided in the embodiments of this application, as shown below. Figure 7 As shown, the anomaly location device 700 includes at least a processor 701 and a computer-readable storage medium 702 configured to store executable instructions, wherein the processor 701 generally controls the overall operation of the anomaly location device. The computer-readable storage medium 702 is configured to store instructions and applications executable by the processor 701, and may also cache data to be processed or processed by various modules in the processor 701 and the anomaly location device 700, and may be implemented using flash memory or random access memory (RAM).

[0130] This application provides a storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the exception location method provided in this application. For example, ... Figure 1 The method shown.

[0131] In some embodiments, the storage medium may be a computer-readable storage medium, such as a ferromagnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or a compact disk-read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0132] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0133] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts within a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files storing one or more modules, subroutines, or code sections). As an example, executable instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0134] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application. It should be understood that "an embodiment" or "one embodiment" mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence number of the above-described processes does not imply the order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments of this application are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not performed.

[0136] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An anomaly localization method, the method comprising: Collect regional video of the target area; Image processing is performed on the video of the region to obtain abnormal video frames corresponding to the abnormal points in the target region; The target area corresponds to a spatial coordinate axis, and the video of the area where the abnormal video frame is located has a corresponding video acquisition point, and the video acquisition point has acquisition coordinates. Feature extraction is performed on the abnormal video frame to obtain the abnormal feature points corresponding to the abnormal video frame; By tracing the abnormal feature points through rays emitted from the acquisition coordinates, the abnormal feature points are mapped to the intersection of the regional model of the target area, so as to locate the abnormal points in the regional model. The target region has multiple regional feature points and multiple video acquisition points at different locations, each video acquisition point corresponding to a acquisition coordinate, and regional video corresponding to the target region is acquired at each of the video acquisition points; the method further includes: In multiple regional videos, at least two regional videos corresponding to each regional feature point are determined, and at least two regional video frames corresponding to the at least two regional videos are determined. Determine the feature coordinates of each region feature point in each of the at least two region video frames; Calculate the three-dimensional coordinates of each feature point in the region based on the acquired coordinates and the feature coordinates; The region model is obtained by surface triangulation based on the three-dimensional coordinates corresponding to each feature point in the region.

2. The method according to claim 1, wherein performing image processing on the regional video to obtain abnormal video frames corresponding to abnormal points in the target region includes: The region video is feature-labeled to obtain the reference video frame corresponding to the region video; Each video frame of the video in the region and the reference video frame are processed to obtain the target grayscale image and the reference grayscale image respectively. The target grayscale image and the reference grayscale image are subjected to similarity processing, and the target grayscale image with a similarity lower than a preset threshold is determined as an abnormal grayscale image; The video frame corresponding to the abnormal grayscale image is identified as the abnormal video frame.

3. The method according to claim 1, wherein the region video where the abnormal video frame is located corresponds to a reference video frame, and the reference video frame is obtained by feature calibration of the region video; The step of extracting features from the abnormal video frames to obtain the abnormal feature points corresponding to the abnormal video frames includes: The regions in the abnormal video frames that are different from the reference video frames are identified as abnormal regions. Edge feature extraction and region shape feature extraction are performed on the abnormal region to obtain the region feature points of the abnormal region, and the region feature points are determined as the abnormal feature points.

4. The method according to claim 1, further comprising: Obtain the vertex coordinates of each surface triangle on the surface of the region model after surface triangulation, the region video corresponding to each surface triangle, and the acquisition coordinates of the video acquisition point corresponding to the region video; Based on the acquired coordinates and the vertex coordinates, determine the texture coordinates corresponding to the vertex coordinates in the regional video; Based on the texture coordinates, determine the texture image corresponding to the texture coordinates in the regional video; The surface triangle is rendered based on the texture image to obtain a real-world region model; wherein the real-world region model is a model capable of representing the real-world image of the target region.

5. The method according to claim 1, The step of mapping the abnormal feature points onto a region model of the target region to locate the abnormal points in the region model includes: Determine the first abnormal coordinates of the abnormal feature point in the abnormal video frame; Based on the collected coordinates and the first abnormal coordinates, the abnormal feature points are mapped onto the regional model of the target area to locate the abnormal points in the regional model.

6. The method according to claim 1, wherein the surface of the region model is composed of a plurality of surface triangles; The method further includes: After mapping the abnormal feature points to the region model, at least one surface triangle containing the abnormal feature points and the vertex coordinates of the at least one surface triangle are determined. Based on the acquisition coordinates and the vertex coordinates, determine the abnormal image corresponding to the at least one surface triangle in the abnormal video frame; The at least one surface triangle is rendered based on the anomalous image to display the anomalous image on the surface of the region model.

7. An anomaly location device, the device comprising: The acquisition module is used to acquire regional video of the target area; The image processing module is used to perform image processing on the video of the region to obtain abnormal video frames corresponding to abnormal points in the target region; the target region corresponds to a spatial coordinate axis, and the video of the region where the abnormal video frame is located has a corresponding video acquisition point, and the video acquisition point has acquisition coordinates; The feature extraction module is used to extract features from the abnormal video frame to obtain the abnormal feature points corresponding to the abnormal video frame. An anomaly localization module is used to map the anomaly feature points to the intersection of the target region's regional model using rays emanating from the acquisition coordinates and passing through the anomaly feature points, thereby locating the anomaly points within the regional model; wherein the target region has multiple regional feature points and multiple video acquisition points at different locations, each video acquisition point corresponding to an acquisition coordinate, and regional video corresponding to the target region is acquired at each video acquisition point; the device further includes: The first determining module is used to determine at least two regional videos corresponding to each regional feature point and at least two regional video frames corresponding to the at least two regional videos in multiple regional videos; The second determining module is used to determine the feature coordinates of each region feature point in each region video frame in the at least two region video frames. The calculation module is used to calculate the three-dimensional coordinates of each feature point in the region based on the acquired coordinates and the feature coordinates; The triangulation module is used to obtain the region model by surface triangulation based on the three-dimensional coordinates corresponding to each feature point in the region.

8. An anomaly location device, the device comprising: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the anomaly location method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing executable instructions for causing a processor to execute the executable instructions to implement the anomaly location method according to any one of claims 1 to 6.

Images