Billions of pixel video-based beach safety warning method, device and system

Abstract

The application relates to a beach safety early warning method and device based on a billion-pixel video and a billion-pixel calculation imaging system. The method comprises the following steps: acquiring a billion-pixel video image photographed for a beach safety monitoring area; identifying a marine area in the billion-pixel video image by using a pre-trained instance segmentation model to obtain image coordinates of a plurality of initial contour vertices; identifying a target object in the billion-pixel video image by using a pre-trained target detection model; screening target contour vertices from the plurality of initial contour vertices, wherein a contour line of a target area formed by the target contour vertices and a contour line of an initial area formed by the initial contour vertices satisfy a preset similarity condition; and judging whether the target object is in a dangerous area based on image coordinates of the target object and image coordinates of the target contour vertices, and performing safety early warning in the case that the target object is in the dangerous area. The method can improve the accuracy and timeliness of beach safety early warning.

Description

Technical Field

[0001] This application relates to the field of computational imaging technology, and in particular to a beach safety early warning method, device, and computational imaging system based on video with hundreds of millions of pixels. Background Technology

[0002] To overcome the limitations of a single camera device in achieving both a wide field of view and high resolution, a computational imaging system based on array cameras with hundreds of millions of pixels has emerged. This system acquires multiple local video streams using an array of cameras with multiple lenses, and then employs relevant algorithms to perform computational processing such as stitching and fusion of these local video streams. This can produce ultra-high-definition fused video at the hundreds of millions or even billions of pixels level, which can be applied to technical fields with large-scale ultra-high-definition video requirements, such as video surveillance in the security field, industrial internet, metaverse, and virtual reality technologies.

[0003] Currently, megapixel-level computational imaging systems have been applied to safety monitoring of beaches and coastlines. Staff can monitor the risky behavior of tourists on the beach using megapixel video feeds. If a tourist is detected entering a dangerous area, a safety alarm is issued and security personnel are notified to take safety measures to protect the tourists' lives and property. However, when the beach area to be monitored is large or there are many tourists, staff find it difficult to achieve comprehensive and real-time safety monitoring and cannot issue timely safety warnings. Summary of the Invention

[0004] Therefore, it is necessary to provide a beach safety early warning method, device, and a billion-pixel computational imaging system based on billion-pixel video to address the aforementioned technical problems, which can ensure the accuracy and timeliness of beach safety early warning.

[0005] Firstly, this application provides a beach safety early warning method based on video with hundreds of millions of pixels. The method includes:

[0006] Acquire video images with hundreds of millions of pixels taken in areas monitored for beach security;

[0007] A pre-trained instance segmentation model is used to identify the ocean region in the billion-pixel video image, and the image coordinates of several initial contour vertices of the ocean region are obtained. A pre-trained object detection model is used to identify the target object in the billion-pixel video image, and the image coordinates of the target object are obtained.

[0008] Target contour vertices are selected from the plurality of initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition.

[0009] Based on the image coordinates of the target object and the image coordinates of the target contour vertices, it is determined whether the target object is in a dangerous area, and a safety warning is issued if the target object is in a dangerous area.

[0010] In one embodiment, the step of selecting target contour vertices from the plurality of initial contour vertices includes:

[0011] Two vertices are selected from the plurality of initial contour vertices, and are respectively used as the initial first endpoint and the second endpoint;

[0012] Calculate the distances between each of the other vertices between the first endpoint and the second endpoint and the line connecting the first endpoint and the second endpoint, and determine the maximum distance.

[0013] If the maximum distance is greater than the distance threshold, then the vertex corresponding to the maximum distance is taken as the new second endpoint corresponding to the first endpoint and the new first endpoint corresponding to the second endpoint. For each pair of first endpoints and second endpoints, the step of calculating the distance between the other vertices between the first endpoint and the second endpoint and the line connecting the first endpoint and the second endpoint is returned.

[0014] If the maximum distance is not greater than the distance threshold, then all other vertices between the first endpoint and the second endpoint are removed, and the first endpoint and the second endpoint are determined as the target contour vertices.

[0015] In one embodiment, the method further includes:

[0016] Based on the perimeter L of the contour line of the initial region formed by the initial contour vertices and the preset scaling factor σ, the distance threshold d is calculated according to the following formula. TH :

[0017] d TH =σ·L.

[0018] In one embodiment, the method further includes:

[0019] For each pair of first endpoints and second endpoints, a distance coefficient λ is determined that matches the position information of the first endpoints and second endpoints. The magnitude of the distance coefficient λ is negatively correlated with the spatial distance between the line connecting the first endpoints and the second endpoints and the camera.

[0020] Based on the perimeter L of the contour line of the initial region formed by each of the initial contour vertices, the preset scaling factor σ, and the matching distance factor λ, the distance threshold d is calculated according to the following formula. TH :

[0021] d TH=λ·σ·L.

[0022] In one embodiment, determining the distance coefficient λ that matches the location information of the first endpoint and the second endpoint includes:

[0023] Based on the image coordinates of the first endpoint and the second endpoint, determine the distance between the midpoint of the line connecting the first endpoint and the second endpoint and the lower boundary of the billion-pixel video image.

[0024] The ratio of the height of the billion-pixel video image to the distance between the midpoint of the connecting line and the lower boundary of the billion-pixel video image is determined as the distance coefficient λ that matches the position information of the first endpoint and the second endpoint.

[0025] In one embodiment, determining the distance coefficient λ that matches the location information of the first endpoint and the second endpoint includes:

[0026] Based on the geographical locations of the first endpoint and the second endpoint, determine the first spatial distance and the second spatial distance of the first endpoint and the second endpoint relative to the camera, respectively, and determine the average value of the first spatial distance and the second spatial distance as the spatial distance of the line connecting the first endpoint and the second endpoint relative to the camera;

[0027] The minimum spatial distance is determined from the spatial distances of each of the initial contour vertices relative to the camera;

[0028] The ratio of the minimum spatial distance to the spatial distance between the line connecting the first endpoint and the second endpoint and the camera is determined as the distance coefficient λ that matches the position information of the first endpoint and the second endpoint.

[0029] In one embodiment, determining whether the target object is in a danger zone based on the image coordinates of the target object and the image coordinates of the target contour vertices includes:

[0030] Based on the image coordinates of the target object and the image coordinates of the target contour vertices, calculate the intersection area of ​​the target region formed by the detection box of the target object and the target contour vertices;

[0031] The size of the intersecting area is used to determine whether the target object is in a dangerous area.

[0032] In one embodiment, determining whether the target object is in a danger zone based on the image coordinates of the target object and the image coordinates of the target contour vertices includes:

[0033] The target region formed by the vertices of the target contour is determined based on the image coordinates of the vertices of the target contour.

[0034] Determine whether the center point of the lower boundary of the detection box of the target object is located within the target area;

[0035] If the center point of the lower boundary of the detection frame is located within the target area, then the target object is determined to be in a dangerous area.

[0036] In one embodiment, selecting two vertices from the plurality of initial contour vertices as the initial first endpoint and second endpoint, respectively, includes:

[0037] Based on the image coordinates of the initial contour vertices, calculate the distance between any two adjacent vertices in the initial contour.

[0038] Two adjacent vertices whose spacing meets the preset conditions are respectively used as the initial first endpoint and the second endpoint.

[0039] In one embodiment, the step of using two adjacent vertices whose spacing satisfies a preset condition as the initial first endpoint and second endpoint, respectively, includes:

[0040] The two adjacent vertices with the largest spacing are taken as the initial first and second endpoints, respectively.

[0041] In one embodiment, the step of using two adjacent vertices whose spacing satisfies a preset condition as the initial first endpoint and second endpoint, respectively, includes:

[0042] The two adjacent vertices with the smallest spacing are taken as the initial first and second endpoints, respectively.

[0043] The step of determining the first endpoint and the second endpoint as target contour vertices includes:

[0044] Other first and second endpoints besides the initial first and second endpoints are identified as target contour points;

[0045] The midpoint of the line connecting the initial first endpoint and the second endpoint is determined as the vertex of the target contour.

[0046] Secondly, this application also provides a beach safety early warning device based on video with hundreds of millions of pixels.

[0047] The device includes:

[0048] The acquisition module is used to acquire video images with hundreds of millions of pixels taken of the beach security monitoring area;

[0049] The first recognition module is used to identify the ocean region in the billion-pixel video image using a pre-trained instance segmentation model, and obtain the image coordinates of several initial contour vertices of the ocean region.

[0050] The second recognition module is used to identify target objects in the billion-pixel video image using a pre-trained target detection model, and obtain the image coordinates of the target objects.

[0051] A filtering module is used to filter out target contour vertices from the plurality of initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition.

[0052] The judgment module is used to determine whether the target object is in a dangerous area based on the image coordinates of the target object and the image coordinates of the target contour vertices, and to issue a safety warning if the target object is in a dangerous area.

[0053] Thirdly, this application also provides a computational imaging system with hundreds of millions of pixels. The system includes an array of cameras and a server;

[0054] The array camera is used to capture multiple local videos of the beach safety monitoring area using multiple local cameras.

[0055] The server is used to stitch and merge the multiple local videos to obtain a video with hundreds of millions of pixels;

[0056] The server is also used to identify ocean regions in a video image with hundreds of millions of pixels using a pre-trained instance segmentation model, obtain image coordinates of several initial contour vertices of the ocean region, and identify target objects in the video image with a pre-trained target detection model, obtain image coordinates of the target objects; select target contour vertices from the several initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition; and determine whether the target object is in a dangerous area based on the image coordinates of the target object and the image coordinates of the target contour vertices, and generate safety warning information if the target object is in a dangerous area.

[0057] The aforementioned beach safety early warning method, device, and computational imaging system based on billion-pixel video images identify marine regions in billion-pixel video images using an instance segmentation model. It outputs the image coordinates of several initial contour vertices of the marine region, employs a target detection model to identify monitored targets, and further thins the initial contour vertices to obtain target contour vertices. Then, it determines whether the monitored target is in a danger zone based on whether it is located within the target region formed by the target contour vertices, providing timely safety warnings. The contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a similarity condition, ensuring the accuracy of the predicted danger zones while reducing the amount of data used for transmission and computation, thus improving the efficiency of safety early warnings. Therefore, this solution can improve the accuracy and timeliness of beach safety early warnings. Attached Figure Description

[0058] Figure 1 This is an application environment diagram of a beach safety early warning method based on videos with hundreds of millions of pixels, as shown in one embodiment.

[0059] Figure 2 This is a flowchart illustrating a beach safety early warning method based on video with hundreds of millions of pixels in one embodiment.

[0060] Figure 3a , Figure 3b These are schematic diagrams of the initial contour line and the target contour line, respectively.

[0061] Figure 4 This is a structural block diagram of a beach safety early warning device in one embodiment. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0063] The beach safety early warning method based on billion-pixel videos provided in this application can be applied to, for example... Figure 1The application environment is shown. The billion-pixel computational imaging system 100 includes an array camera 102 and a server 104. The array camera 102 includes multiple local lenses (or sub-cameras) with different shooting angles, such as common 3x5, 3x6, or 1x10 array cameras. Each lens captures a local video (usually 4K video) to acquire images of a local area within the shooting scene (such as a beach security monitoring area). Typically, the local video images captured by two lenses with adjacent fields of view have overlapping areas. The server 104 can use image stitching algorithms or a pre-deployed image stitching fusion model to stitch and fuse the various local video images, obtaining a panoramic fused image with a larger field of view. This fused video (which can be called a billion-pixel video) is then sent to the display terminal 106 for playback. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers. The display terminal 106 can be implemented using various personal computers, laptops, smartphones, tablets, smart displays, IoT devices, portable wearable devices, and other electronic devices with display units.

[0064] In one embodiment, such as Figure 2 As shown, a beach safety early warning method based on videos with hundreds of millions of pixels is provided. This method can be applied to... Figure 1 The server shown is part of a multi-megapixel computational imaging system. In this embodiment, the method includes the following steps:

[0065] Step 202: Acquire video images with hundreds of millions of pixels taken of the beach security monitoring area.

[0066] In implementation, array cameras can be installed on the coast to capture real-time video of the beach safety monitoring area. The server can include a fusion service module and a detection service module. The fusion service module can acquire multiple local video streams captured by the array cameras on the beach safety monitoring area, and use image stitching algorithms or pre-deployed image stitching and fusion models to stitch and fuse the local video frames to generate a video with hundreds of millions of pixels. The detection service module can obtain the video with hundreds of millions of pixels from the fusion service module to identify dangerous areas and target objects (such as tourists).

[0067] Step 204: Use a pre-trained instance segmentation model to identify the ocean region in the video image with hundreds of millions of pixels, obtain the image coordinates of several initial contour vertices of the ocean region, and use a pre-trained object detection model to identify the target object in the video image with hundreds of millions of pixels, obtain the image coordinates of the target object.

[0068] In implementation, an instance segmentation model can be pre-trained using hundreds of millions of pixel sample images taken of the beach safety monitoring area. This model is used to detect and segment ocean areas (dangerous areas) and land areas (safe areas) in the images, outputting the image coordinates (pixel coordinates) of the initial contour vertices of the ocean area. Connecting these vertices sequentially yields the initial contour line of the ocean area. To predict the ocean area as accurately as possible, the instance segmentation model outputs a large number of initial contour vertices, containing more contour line details, resulting in high prediction accuracy. Figure 3a As shown in the image, the green outline is the initial outline formed by the initial outline vertices (a total of 205). Understandably, to improve annotation and model training efficiency, the distant sky can be labeled as a danger zone (merged with the ocean area) for model training. The corresponding detected danger zones will include the sky area (which is also an area where tourists should not be).

[0069] In addition, a target detection model can be trained in advance using sample images containing target objects such as pedestrians to identify target objects in video images with hundreds of millions of pixels and obtain the image coordinates of the detection box corresponding to the target object.

[0070] Step 206: Select target contour vertices from a number of initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition.

[0071] In implementation, the area enclosed by the initial contour line is usually a complex polygonal region, especially the boundary line between the ocean and the beach / reef (coastline). These areas have complex and irregular shapes, involving many vertices, resulting in a large number of vertices and edges in the enclosed polygonal region. The large amount of contour vertex data presents several problems. First, the detection service module needs to transmit the danger zone identification results (image coordinates of each initial contour vertex) to other modules (such as the fusion service module or other intermediate service modules) for subsequent warning judgment. Too much data will affect data transmission efficiency or require higher bandwidth, leading to high data transmission costs. Second, determining whether a target object is in a danger zone usually involves calculating the intersection area of ​​the target object's detection box and the danger zone (i.e., the intersection area of ​​two polygons), or determining whether the target object's feature points (such as the center point of the detection box, the center point of the bottom edge, or feature points at other locations) are located in the danger zone—that is, determining whether a point is inside the polygon. If the polygon has many vertices and edges, the calculation time increases significantly, seriously affecting the efficiency of safety warnings and causing delays in alarms and safety rescues.

[0072] Therefore, the detection service module can further filter out target contour vertices from several initial contour vertices. Furthermore, the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition. This allows for ensuring the accuracy of the segmented hazardous areas while simultaneously thinning the polygonal vertices of the hazardous areas, simplifying the polygons, reducing data volume, and improving the efficiency of safety warnings. Figure 3b As shown in the figure, the green outline is the target outline composed of 21 target outline vertices. It can be seen that although the number of target outline vertices is much smaller than that of the initial outline vertices, the target outline and the initial outline have a high degree of similarity, which can accurately describe the location information of the danger zone.

[0073] Step 208: Based on the image coordinates of the target object and the image coordinates of the target contour vertices, determine whether the target object is in a dangerous area, and issue a safety warning if the target object is in a dangerous area.

[0074] In practice, the fusion service module can determine whether the target object is in a dangerous area based on the image coordinates of the target object output by the detection service module and the image coordinates of the predicted target contour vertices of the ocean area. If the target object is detected to be in a dangerous area, a safety warning message can be generated and pushed to the display terminal.

[0075] In one implementation, the fusion service module can calculate the intersection area of ​​the target region formed by the detection box of the target object and the vertices of the target contour based on the image coordinates of the target object and the image coordinates of the target contour vertices. That is, it calculates the intersection area of ​​the rectangle (detection box) and the polygon (target region), and then determines whether the target object is in a dangerous area based on the size of the intersection area. For example, if the intersection area is greater than a preset threshold, or the ratio of the intersection area to the detection box area is greater than a preset threshold, then the target object can be determined to be in a dangerous area.

[0076] In another implementation, the fusion service module can determine the target region formed by the target contour vertices based on the image coordinates of the target contour vertices. It then determines whether the center point of the lower boundary of the target object's detection box is located within this target region. If the center point is within the target region, the target object is considered to be in a danger zone. If the center point is not within the target region, the target object can be considered to be in a safe zone, or other methods can be combined to further determine if it is in a danger zone. If the center point of the lower boundary of the target object's detection box is already in the danger zone, it indicates that the pedestrian's feet have completely entered the danger zone. This method can quickly and accurately detect relatively dangerous situations, issue timely warnings, and improve the efficiency of safety warnings and rescue operations.

[0077] The aforementioned beach safety early warning method based on billion-pixel video images identifies marine regions in the billion-pixel video images using an instance segmentation model, outputting the image coordinates of several initial contour vertices of the marine region. A target detection model is then used to identify the monitored target, and the initial contour vertices are further thinned to obtain the target contour vertices. Finally, whether the monitored target is located within the target region formed by the target contour vertices is determined to indicate whether the target is in a dangerous area, and a timely safety warning is issued. The contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a similarity condition, which ensures the accuracy of the predicted dangerous areas while reducing the amount of data used for transmission and computation, thus improving the efficiency of the safety early warning. Therefore, this scheme can improve the accuracy and timeliness of beach safety early warnings.

[0078] In one embodiment, the process of filtering target contour vertices in step 206 includes the following steps:

[0079] Step 2062: Select two vertices from a number of initial contour vertices, and use them as the first endpoint and the second endpoint, respectively.

[0080] In implementation, two adjacent vertices can be randomly selected from a pool of initial contour vertices, or other selection strategies can be used, to serve as the initial first endpoint and second endpoint, respectively. The remaining vertices are the midpoints between these pairs. Since the contour vertices have a connection order, the connection proceeds from the initial first endpoint to the midpoint, then to the initial second endpoint, and finally back to the initial first endpoint, thus forming a complete initial contour line. That is, if the connection order of the initial first endpoint is 1, then the order of the initial second endpoints is -1 or n (a total of n vertices).

[0081] Step 2064: Calculate the distances between each of the other vertices between the first endpoint and the second endpoint and the line connecting the first endpoint and the second endpoint, and determine the maximum distance.

[0082] In implementation, for each other vertex in the connection sequence between the first and second endpoints, the perpendicular distance between that vertex and the line connecting the first and second endpoints can be calculated. Then, among the perpendicular distances corresponding to each vertex, the maximum distance is found, i.e., the vertex farthest from the line connecting that vertex and the first endpoint is found.

[0083] Specifically, the perpendicular distance can be obtained from the area of ​​the triangle formed by the first endpoint, the second endpoint, and the vertex, as well as the length of the line segment connecting the first endpoint and the second endpoint.

[0084] Assume the first and second endpoints are denoted as points A and B, and point C is a vertex between these two endpoints. The image coordinates of these three points are (A, B, C ... x A y ), (Bx B y ), (C x C y ).

[0085] (1) Use the cross product method to calculate the area S of the triangle formed by the three points.

[0086]

[0087] (2) Calculate the length of the line segment connecting points A and B.

[0088]

[0089] (3) Calculate the perpendicular distance from point C to the line (straight line) connecting point C and AB.

[0090] d vertical =2S / d AB

[0091] Step 2066: If the maximum distance is greater than the distance threshold, then the vertex corresponding to the maximum distance is taken as the new second endpoint corresponding to the first endpoint and the new first endpoint corresponding to the second endpoint. For each pair of first endpoints and second endpoints, return to the step of calculating the distance between the other vertices between the first endpoint and the second endpoint and the line connecting the first endpoint and the second endpoint.

[0092] In implementation, a distance threshold can be preset or determined according to other rules. Then, the maximum distance is compared with the distance threshold. If the maximum distance is greater than the distance threshold, the vertex corresponding to the maximum distance is retained. At this time, the original first endpoint, second endpoint, and retained point can form two pairs of points: the first endpoint and the retained point, and the retained point and the second endpoint. Since the connection order of the retained point is between the original first endpoint and the second endpoint, in the two newly formed point pairs, the retained point can be used as the second endpoint corresponding to the original first endpoint, and the first endpoint corresponding to the original second endpoint.

[0093] Then, for each pair of first and second endpoints, step 2064 is recursively executed to calculate the distances between the lines connecting all other vertices in the connection order between the first and second endpoints and the pair of first and second endpoints, and the maximum distance corresponding to the pair of first and second endpoints is determined. If the maximum distance is greater than a preset threshold, the retained point is combined with the pair of first and second endpoints to form two new point pairs.

[0094] Step 2068: If the maximum distance is not greater than the distance threshold, remove all other vertices between the first endpoint and the second endpoint, and determine the first endpoint and the second endpoint as the target contour vertices.

[0095] In implementation, if the maximum distance is not greater than a distance threshold, all vertices connected in sequence between the first and second endpoints of the pair are removed, and the first and second endpoints of the pair are determined as the target contour vertices. This removes a large amount of vertex data that has little impact on the contour shape. The contour line formed by connecting the selected target contour vertices satisfies the similarity condition to the contour line formed by connecting the initial contour vertices, ensuring the accuracy of the predicted ocean area. Furthermore, the reduced data volume improves the efficiency of data transmission and subsequent security assessments, thereby enhancing the efficiency of security early warning.

[0096] In one embodiment, step 2062 can employ the following selection strategy to select the initial first endpoint and the second endpoint: based on the image coordinates of the initial contour vertices, calculate the distance between any two adjacent vertices in the initial contour vertices, and then use the two adjacent vertices whose distance satisfies the preset condition as the initial first endpoint and the second endpoint, respectively.

[0097] In one implementation, the two adjacent vertices with the largest distance can be used as the initial first and second endpoints, respectively. For two adjacent vertices with a large distance between them, removing one of them could significantly impact the contour line; therefore, it is necessary to retain the two adjacent vertices with the largest distance between them. Thus, selecting the two adjacent vertices with the largest distance between them as the initial first and second endpoints, and retaining them as the target contour vertices, can ensure the accuracy of the hazardous area contour line while minimizing the amount of data and improving early warning efficiency.

[0098] In another implementation, the two adjacent vertices with the smallest distance can be used as the initial first and second endpoints, respectively. Correspondingly, in step 2068, it can be determined whether the first endpoint of the currently processed point pair is the initial first or second endpoint (based on the image coordinates of each vertex). If not, the first endpoint is determined as the target contour vertex. Similarly, it can be determined whether the second endpoint of the currently processed point pair is the initial first or second endpoint. If not, the second endpoint is determined as the target contour vertex. For the initial first or second endpoint, the midpoint of the line connecting the initial first and second endpoints can be determined as the target contour vertex. For two adjacent vertices with a small distance, removing one vertex or merging the two vertices has little impact on the contour line. Therefore, the two adjacent vertices with the smallest distance can be used as the initial first and second endpoints, and then the initial first and second endpoints can be merged. Alternatively, either one can be chosen as the target contour vertex. This ensures the accuracy of the dangerous area contour line while minimizing data volume and improving early warning efficiency.

[0099] In one embodiment, the distance threshold can be determined according to the following rules to ensure its reasonableness and balance the accuracy and efficiency of the safety warning. Specifically, the distance threshold d can be calculated based on the perimeter L of the contour line of the initial region formed by each initial contour vertex and a preset scaling factor σ, according to the following formula. TH :

[0100] d TH =σ·L.

[0101] The perimeter L of the contour line can be calculated based on the image coordinates of each initial contour vertex. The scaling factor σ is a parameter less than 1 and can be preset, for example, it can be set to 0.001 to 0.005.

[0102] In another embodiment, the distance threshold can be determined according to the following rule: For each pair of first and second endpoints, a distance coefficient λ matching the position information of the first and second endpoints is determined, wherein the magnitude of the distance coefficient λ is negatively correlated with the spatial distance between the line connecting the first and second endpoints and the camera. Based on the perimeter L of the contour line of the initial region formed by each initial contour vertex, the preset scaling factor σ, and the matching distance coefficient λ, the distance threshold d is calculated according to the following formula. TH :

[0103] d TH =λ·σ·L.

[0104] In implementation, because a billion-pixel computational imaging system can balance a large field of view and high definition, a billion-pixel video image typically covers a large field of view, which may include monitoring areas closer to the camera and monitoring areas farther away from the camera. For areas of the same spatial size, the farther away from the camera, the fewer pixels they occupy in the image. Therefore, it is necessary to preserve as much detail information as possible about the coastline (outline) of distant monitoring areas to ensure the accuracy of security detection and early warning. Thus, it is necessary to adjust the thinning degree of the outline vertices according to the distance of the monitoring area. In this embodiment, by setting a distance coefficient λ that matches the position information of the first and second endpoints, the thinning degree of the outline vertices of relatively distant ocean areas is weaker, and the thinning degree of the outline vertices of relatively nearby ocean areas is relatively stronger. This reduces the amount of data while preserving necessary detail information, thus balancing the accuracy and efficiency of early warning.

[0105] In one implementation, the process of determining the matching distance coefficient λ includes: determining the distance between the midpoint of the line connecting the first endpoint and the second endpoint and the lower boundary of the billion-pixel video image based on the image coordinates of the first endpoint and the second endpoint; and determining the ratio of the height of the billion-pixel video image to the distance between the midpoint of the line connecting the first endpoint and the lower boundary of the billion-pixel video image as the distance coefficient λ that matches the position information of the first endpoint and the second endpoint.

[0106] In implementation, for each pair of first and second endpoints, the distance between the midpoint of the line connecting the first and second endpoints and the lower boundary of the billion-pixel video image can be calculated. For example, based on the image ordinates of the first and second endpoints and the height of the billion-pixel video image (such as the ordinate of the last row of pixels), the distances (pixel distances) between the first and second endpoints and the lower boundary of the image can be calculated. The average of these two distances is the distance between the midpoint of the line connecting the two endpoints and the lower boundary. Then, dividing the image height by the distance between the midpoint of the line connecting the two endpoints yields the distance coefficient λ that matches the positional information of the pair of first and second endpoints. Typically, in images captured by array cameras, the area at the bottom of the image is closer to the array camera, while the area at the top of the image is farther away. Therefore, the distance between the midpoint of the line connecting the first and second endpoints and the lower boundary of the image reflects the distance of the area where the two endpoints are located relative to the camera. Thus, the above method can be used to determine the matching distance coefficient λ, balancing the accuracy and efficiency of the warning.

[0107] In one implementation, the process of determining the matching distance coefficient λ includes: determining the first spatial distance and the second spatial distance of the first endpoint and the second endpoint relative to the camera based on their geographical locations, and determining the average of the first spatial distance and the second spatial distance as the spatial distance of the line connecting the first endpoint and the second endpoint relative to the camera; determining the minimum spatial distance from the spatial distances of each initial contour vertex relative to the camera; and determining the ratio of the minimum spatial distance to the spatial distance of the line connecting the first endpoint and the second endpoint relative to the camera as the distance coefficient λ that matches the position information of the first endpoint and the second endpoint.

[0108] In implementation, a pre-established coordinate mapping relationship can be used to convert the image coordinates of each initial contour vertex into geographic coordinates. This allows the calculation of the spatial distance between each vertex and the camera's installation location based on their geographic coordinates. Then, the average spatial distance between the first and second endpoints can be determined as the spatial distance between the line connecting the first and second endpoints and the camera, reflecting the distance between the areas of the first and second endpoints and the camera. Next, the minimum spatial distance can be determined from the spatial distances of each initial contour vertex relative to the camera. The ratio of this minimum spatial distance to the spatial distance between the line connecting the first and second endpoints and the camera is determined as the distance coefficient λ that matches the positional information of the first and second endpoints. Therefore, the greater the spatial distance, the smaller the distance coefficient λ, and the corresponding distance threshold d... TH The smaller the value, the weaker the thinning effect, and the more necessary vertex and detail information is retained, thus balancing the accuracy and efficiency of the early warning.

[0109] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0110] Based on the same inventive concept, this application also provides a beach safety warning device based on hundreds of millions of pixel videos for implementing the aforementioned beach safety warning method based on hundreds of millions of pixel videos. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the beach safety warning device based on hundreds of millions of pixel videos provided below can be found in the limitations of the beach safety warning method based on hundreds of millions of pixel videos described above, and will not be repeated here.

[0111] In one embodiment, such as Figure 4 As shown, a beach safety early warning device 400 based on video with hundreds of millions of pixels is provided, including: an acquisition module 401, a first identification module 402, a second identification module 403, a filtering module 404, and a judgment module 405, wherein:

[0112] The acquisition module 401 is used to acquire video images with hundreds of millions of pixels taken of the beach security monitoring area.

[0113] The first recognition module 402 is used to identify the ocean region in the billion-pixel video image using a pre-trained instance segmentation model, and obtain the image coordinates of several initial contour vertices of the ocean region.

[0114] The second recognition module 403 is used to identify target objects in the video image with hundreds of millions of pixels using a pre-trained target detection model, and obtain the image coordinates of the target objects.

[0115] The filtering module 404 is used to filter out target contour vertices from the plurality of initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition.

[0116] The judgment module 405 is used to determine whether the target object is in a dangerous area based on the image coordinates of the target object and the image coordinates of the target contour vertices, and to issue a safety warning if the target object is in a dangerous area.

[0117] In one embodiment, the filtering module 404 is specifically configured to: select two vertices from the plurality of initial contour vertices, respectively as the initial first endpoint and the second endpoint; calculate the distances between the other vertices between the first endpoint and the second endpoint and the lines connecting the first endpoint and the second endpoint, and determine the maximum distance; if the maximum distance is greater than a distance threshold, then the vertex corresponding to the maximum distance is used as the new second endpoint corresponding to the first endpoint and the new first endpoint corresponding to the second endpoint, and for each pair of first endpoints and second endpoints, return to the step of calculating the distances between the other vertices between the first endpoint and the second endpoint and the lines connecting the first endpoint and the second endpoint; if the maximum distance is not greater than a distance threshold, then remove the other vertices between the first endpoint and the second endpoint, and determine the first endpoint and the second endpoint as the target contour vertices.

[0118] In one embodiment, the filtering module 404 is further configured to: calculate a distance threshold d according to the perimeter L of the contour line of the initial region formed by each of the initial contour vertices and a preset scaling factor σ, using the following formula. TH :

[0119] d TH =σ·L.

[0120] In one embodiment, the filtering module 404 is further configured to: for each pair of first endpoints and second endpoints, determine a distance coefficient λ that matches the position information of the first endpoints and second endpoints, wherein the magnitude of the distance coefficient λ is negatively correlated with the spatial distance between the line connecting the first endpoints and second endpoints and the camera; and calculate a distance threshold d according to the following formula based on the perimeter L of the contour line of the initial region formed by each of the initial contour vertices, a preset scaling factor σ, and the matching distance coefficient λ. TH :

[0121] d TH =λ·σ·L.

[0122] In one embodiment, the filtering module 404 is further configured to: determine the distance between the midpoint of the line connecting the first endpoint and the second endpoint and the lower boundary of the billion-pixel video image based on the image coordinates of the first endpoint and the second endpoint; and determine the ratio of the height of the billion-pixel video image to the distance between the midpoint of the line connecting the first endpoint and the lower boundary of the billion-pixel video image as a distance coefficient λ that matches the position information of the first endpoint and the second endpoint.

[0123] In one embodiment, the filtering module 404 is further configured to: determine a first spatial distance and a second spatial distance between the first endpoint and the second endpoint and the camera, respectively, based on the geographical locations of the first endpoint and the second endpoint; determine the average of the first spatial distance and the second spatial distance as the spatial distance between the line connecting the first endpoint and the second endpoint and the camera; determine the minimum spatial distance from the spatial distances between each of the initial contour vertices and the camera; and determine the ratio of the minimum spatial distance to the spatial distance between the line connecting the first endpoint and the second endpoint and the camera as a distance coefficient λ that matches the position information of the first endpoint and the second endpoint.

[0124] In one embodiment, the determination module 405 is specifically used to calculate the intersection area of ​​the target region formed by the detection box of the target object and the target contour vertices based on the image coordinates of the target object and the image coordinates of the target contour vertices; and to determine whether the target object is in a danger zone based on the size of the intersection area.

[0125] In one embodiment, the determination module 405 is specifically used to determine the target region formed by the target contour vertices based on the image coordinates of the target contour vertices; determine whether the center point of the lower boundary of the detection box of the target object is located within the target region; if the center point of the lower boundary of the detection box is located within the target region, then determine that the target object is in a danger zone.

[0126] In one embodiment, the filtering module 404 is further configured to: calculate the distance between any two adjacent vertices in the initial contour vertices based on the image coordinates of the initial contour vertices; and use the two adjacent vertices whose distance satisfies a preset condition as the initial first endpoint and the second endpoint, respectively.

[0127] In one embodiment, the filtering module 404 is further configured to: use the two adjacent vertices with the largest spacing as the initial first endpoint and the second endpoint, respectively.

[0128] In one embodiment, the filtering module 404 is further configured to: designate the two adjacent vertices with the smallest spacing as the initial first endpoint and the second endpoint, respectively; designate the other first endpoints and second endpoints besides the initial first endpoints and the second endpoints as target contour points; and designate the midpoint of the line connecting the initial first endpoints and the second endpoints as the target contour vertex.

[0129] The various modules in the aforementioned beach safety early warning device based on billion-pixel video can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0130] In one embodiment, a 100-megapixel computational imaging system is also provided, the system comprising an array of cameras and a server.

[0131] The array camera is used to capture multiple local videos of the beach safety monitoring area using multiple local cameras.

[0132] The server is used to stitch and merge the multiple local videos to obtain a video with hundreds of millions of pixels.

[0133] The server is also used to identify ocean regions in a video image with hundreds of millions of pixels using a pre-trained instance segmentation model, obtain image coordinates of several initial contour vertices of the ocean region, and identify target objects in the video image with a pre-trained target detection model, obtain image coordinates of the target objects; select target contour vertices from the several initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition; and determine whether the target object is in a dangerous area based on the image coordinates of the target object and the image coordinates of the target contour vertices, and generate safety warning information if the target object is in a dangerous area.

[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0135] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A beach safety early warning method based on videos with hundreds of millions of pixels, characterized in that, The method includes: Acquire video images with hundreds of millions of pixels taken in areas monitored for beach security; A pre-trained instance segmentation model is used to identify the ocean region in the billion-pixel video image, and the image coordinates of several initial contour vertices of the ocean region are obtained. A pre-trained object detection model is used to identify the target object in the billion-pixel video image, and the image coordinates of the target object are obtained. Target contour vertices are selected from the plurality of initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy the similarity condition. Based on the image coordinates of the target object and the image coordinates of the target contour vertices, determine whether the target object is in a dangerous area, and issue a safety warning if the target object is in a dangerous area. The step of selecting target contour vertices from the plurality of initial contour vertices includes: Two vertices are selected from the plurality of initial contour vertices, and are respectively used as the initial first endpoint and the second endpoint; Calculate the distances between each of the other vertices between the first endpoint and the second endpoint and the line connecting the first endpoint and the second endpoint, and determine the maximum distance. If the maximum distance is greater than the distance threshold, then the vertex corresponding to the maximum distance is taken as the new second endpoint corresponding to the first endpoint and the new first endpoint corresponding to the second endpoint. For each pair of first endpoints and second endpoints, the step of calculating the distance between the other vertices between the first endpoint and the second endpoint and the line connecting the first endpoint and the second endpoint is returned. If the maximum distance is not greater than the distance threshold, then remove all other vertices between the first endpoint and the second endpoint, and determine the first endpoint and the second endpoint as the target contour vertices; The method further includes: For each pair of first and second endpoints, determine the distance coefficient that matches the position information of the first and second endpoints. Among them, distance coefficient The size of the distance is negatively correlated with the spatial distance between the line connecting the first endpoint and the second endpoint and the camera. Based on the perimeter L of the contour line of the initial region formed by the initial contour vertices and the preset scaling factor... and the matching distance coefficient Calculate the distance threshold using the following formula. : ； Wherein, the distance coefficient for determining the position information matching the first endpoint and the second endpoint is... This includes: determining the distance between the midpoint of the line connecting the first endpoint and the second endpoint and the lower boundary of the billion-pixel video image based on the image coordinates of the first endpoint and the second endpoint; and determining the ratio of the height of the billion-pixel video image to the distance between the midpoint of the line connecting the first endpoint and the lower boundary of the billion-pixel video image as a distance coefficient that matches the position information of the first endpoint and the second endpoint. Alternatively, based on the geographical locations of the first and second endpoints, determine the first and second spatial distances relative to the camera, respectively, and determine the average of the first and second spatial distances as the spatial distance of the line connecting the first and second endpoints relative to the camera; determine the minimum spatial distance from the spatial distances of each of the initial contour vertices relative to the camera; and determine the ratio of the minimum spatial distance to the spatial distance of the line connecting the first and second endpoints relative to the camera as the distance coefficient matching the position information of the first and second endpoints. .

2. The method according to claim 1, characterized in that, The step of selecting two vertices from the plurality of initial contour vertices, respectively as the initial first endpoint and the second endpoint, includes: Based on the image coordinates of the initial contour vertices, calculate the distance between any two adjacent vertices in the initial contour. Two adjacent vertices whose spacing meets the preset conditions are respectively used as the initial first endpoint and the second endpoint.

3. The method according to claim 2, characterized in that, The step of using two adjacent vertices whose spacing meets a preset condition as the initial first endpoint and second endpoint, respectively, includes: The two adjacent vertices with the largest spacing are taken as the initial first and second endpoints, respectively.

4. The method according to claim 2, characterized in that, The step of using two adjacent vertices whose spacing meets a preset condition as the initial first endpoint and second endpoint, respectively, includes: The two adjacent vertices with the smallest spacing are taken as the initial first and second endpoints, respectively. The step of determining the first endpoint and the second endpoint as target contour vertices includes: Other first and second endpoints besides the initial first and second endpoints are identified as target contour points; The midpoint of the line connecting the initial first endpoint and the second endpoint is determined as the vertex of the target contour.

5. The method according to claim 1, characterized in that, The step of determining whether the target object is in a danger zone based on the image coordinates of the target object and the image coordinates of the target contour vertices includes: The target region formed by the vertices of the target contour is determined based on the image coordinates of the vertices of the target contour. Determine whether the center point of the lower boundary of the detection box of the target object is located within the target area; If the center point of the lower boundary of the detection frame is located within the target area, then the target object is determined to be in a dangerous area.

6. A beach safety early warning device based on video with hundreds of millions of pixels, characterized in that, The device includes: The acquisition module is used to acquire video images with hundreds of millions of pixels taken of the beach security monitoring area; The first recognition module is used to identify the ocean region in the billion-pixel video image using a pre-trained instance segmentation model, and obtain the image coordinates of several initial contour vertices of the ocean region. The second recognition module is used to identify target objects in the billion-pixel video image using a pre-trained target detection model, and obtain the image coordinates of the target objects. A filtering module is used to filter target contour vertices from a plurality of initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition; wherein, the step of filtering target contour vertices from the plurality of initial contour vertices includes: selecting two vertices from the plurality of initial contour vertices as initial first endpoints and second endpoints respectively; calculating the distances between the other vertices between the first endpoint and the second endpoint and the lines connecting the first endpoint and the second endpoint, and determining the maximum distance; if the maximum distance is greater than a distance threshold, then the vertex corresponding to the maximum distance is taken as the new second endpoint corresponding to the first endpoint and the new first endpoint corresponding to the second endpoint, and for each pair of first endpoints and second endpoints, the step of calculating the distances between the other vertices between the first endpoint and the second endpoint and the lines connecting the first endpoint and the second endpoint is returned; if the maximum distance is not greater than a distance threshold, then the other vertices between the first endpoint and the second endpoint are removed, and the first endpoint and the second endpoint are determined as target contour vertices; The filtering module is also used to determine a distance coefficient that matches the position information of the first endpoint and the second endpoint for each pair of first endpoints and second endpoints. Among them, distance coefficient The size is negatively correlated with the spatial distance between the line connecting the first and second endpoints and the camera; based on the perimeter L of the contour line of the initial region formed by each of the initial contour vertices and a preset scaling factor. and the matching distance coefficient Calculate the distance threshold using the following formula. : Wherein, the distance coefficient for determining the position information matching the first endpoint and the second endpoint... This includes: determining the distance between the midpoint of the line connecting the first endpoint and the second endpoint and the lower boundary of the billion-pixel video image based on the image coordinates of the first endpoint and the second endpoint; and determining the ratio of the height of the billion-pixel video image to the distance between the midpoint of the line connecting the first endpoint and the lower boundary of the billion-pixel video image as a distance coefficient that matches the position information of the first endpoint and the second endpoint. Alternatively, based on the geographical locations of the first and second endpoints, determine the first and second spatial distances relative to the camera, respectively, and determine the average of the first and second spatial distances as the spatial distance of the line connecting the first and second endpoints relative to the camera; determine the minimum spatial distance from the spatial distances of each of the initial contour vertices relative to the camera; and determine the ratio of the minimum spatial distance to the spatial distance of the line connecting the first and second endpoints relative to the camera as the distance coefficient matching the position information of the first and second endpoints. ; The judgment module is used to determine whether the target object is in a dangerous area based on the image coordinates of the target object and the image coordinates of the target contour vertices, and to issue a safety warning if the target object is in a dangerous area.

7. A computational imaging system with hundreds of millions of pixels, characterized in that, The system includes an array of cameras and a server; The array camera is used to capture multiple local videos of the beach safety monitoring area using multiple local cameras. The server is used to stitch and merge the multiple local videos to obtain a video with hundreds of millions of pixels; The server is also used to identify ocean regions in billion-pixel video images using a pre-trained instance segmentation model, obtaining image coordinates of several initial contour vertices of the ocean region, and to identify target objects in the billion-pixel video images using a pre-trained object detection model, obtaining image coordinates of the target objects; to filter target contour vertices from the several initial contour vertices, wherein the contour lines of the target region formed by the target contour vertices and the contour lines of the initial region formed by the initial contour vertices satisfy a preset similarity condition; based on the image coordinates of the target object and the image coordinates of the target contour vertices, to determine whether the target object is in a dangerous area, and to generate safety warning information if the target object is in a dangerous area; wherein, the step of filtering target contour vertices from the several initial contour vertices includes: from the... Two vertices are selected from a plurality of initial contour vertices and designated as the first endpoint and the second endpoint, respectively. The distances between the other vertices between the first endpoint and the second endpoint and the lines connecting the first endpoint and the second endpoint are calculated, and the maximum distance is determined. If the maximum distance is greater than a distance threshold, the vertex corresponding to the maximum distance is designated as the new second endpoint corresponding to the first endpoint and the new first endpoint corresponding to the second endpoint. For each pair of first endpoints and second endpoints, the step of calculating the distances between the other vertices between the first endpoint and the second endpoint and the lines connecting the first endpoint and the second endpoint is returned. If the maximum distance is not greater than the distance threshold, the other vertices between the first endpoint and the second endpoint are removed, and the first endpoint and the second endpoint are determined as the target contour vertices. The server is also configured to determine, for each pair of first endpoints and second endpoints, a distance coefficient matching the location information of the first endpoints and second endpoints. Among them, distance coefficient The size is negatively correlated with the spatial distance between the line connecting the first and second endpoints and the camera; based on the perimeter L of the contour line of the initial region formed by each of the initial contour vertices and a preset scaling factor. and the matching distance coefficient Calculate the distance threshold using the following formula. : Wherein, the distance coefficient for determining the position information matching the first endpoint and the second endpoint... This includes: determining the distance between the midpoint of the line connecting the first endpoint and the second endpoint and the lower boundary of the billion-pixel video image based on the image coordinates of the first endpoint and the second endpoint; and determining the ratio of the height of the billion-pixel video image to the distance between the midpoint of the line connecting the first endpoint and the lower boundary of the billion-pixel video image as a distance coefficient that matches the position information of the first endpoint and the second endpoint. Alternatively, based on the geographical locations of the first and second endpoints, determine the first and second spatial distances relative to the camera, respectively, and determine the average of the first and second spatial distances as the spatial distance of the line connecting the first and second endpoints relative to the camera; determine the minimum spatial distance from the spatial distances of each of the initial contour vertices relative to the camera; and determine the ratio of the minimum spatial distance to the spatial distance of the line connecting the first and second endpoints relative to the camera as the distance coefficient matching the position information of the first and second endpoints. .

Images