Apparatus, method, and program for identifying shooting blind spots and generating additional shooting path based on mesh surface analysis of identified object using drone device

Abstract

The present disclosure provides an apparatus, method, and program for identifying shooting blind spots and generating additional shooting paths based on mesh surface analysis of a subject identified using a drone. A method according to one embodiment of the present disclosure includes the steps of: acquiring a plurality of point cloud scan data corresponding to the subject obtained by photographing the subject through a drone; aligning the plurality of scan data to generate a three-dimensional model of the subject and identifying a plurality of meshes constituting the three-dimensional model; identifying a shooting blind spot of the drone for the subject among the plurality of meshes using a virtual camera in a pre-set coordinate space; and setting an additional shooting path of the drone for the blind spot based on the normal vector of the mesh constituting the identified blind spot.

Description

Technology Field

[0001] The present disclosure relates to an apparatus, method, and program for identifying shooting blind spots and generating additional shooting paths based on mesh surface analysis of a subject identified using a drone. Background Technology

[0002] With the recent advancement of drone technology, aerial photography techniques utilizing drones are being widely utilized across various industrial sectors. Equipped with high-resolution cameras or distance sensors (such as LiDAR and ToF sensors), drones offer greater efficiency and safety compared to traditional human-centered methods in areas like topographic surveying, building exterior inspection, power line management, structural safety checks of bridges and plant facilities, and cultural heritage restoration. Their utility is particularly high due to their ability to precisely capture the exteriors of hard-to-access subjects, such as high-rise structures or large facilities.

[0003] However, during the process of capturing video or 3D point clouds (3D scans) using drones, problems arise where parts of the subject are out of the camera's field of view depending on the shape of the subject, the drone's flight path, the camera's angle of view, and the presence of surrounding obstacles. These areas are generally referred to as blind spots or unscanned areas, and they result in data loss when creating 3D models of the subject or analyzing external defects.

[0004] Such blind spots in imaging cause problems that result in the incomplete restoration of parts of the structure's shape, as well as the degradation of 3D model quality and alignment errors. Particularly in complex curved structures or concave areas, securing a clear view is difficult due to limitations on the drone's flight angle and distance; consequently, the same point may need to be photographed repeatedly, or parts of the area may be completely omitted. These issues reduce work efficiency and undermine the reliability of structural inspections or modeling. Therefore, a technical approach is required to efficiently recognize and minimize blind spots during the drone imaging process. The problem to be solved

[0005] The present disclosure was devised to solve the aforementioned problem and aims to provide an apparatus and method for identifying shooting blind spots and generating additional shooting paths based on mesh surface analysis of a subject identified using a drone.

[0006] However, the problems to be solved in this disclosure are not limited to those mentioned above, and other unmentioned problems may be clearly understood based on the description below. means of solving the problem

[0007] To solve the above-mentioned problem, a method for identifying a shooting blind spot and generating an additional shooting path based on mesh surface analysis of a subject identified using a drone according to one embodiment of the present disclosure comprises: a step of acquiring a plurality of point cloud scan data corresponding to the subject obtained by shooting the subject through a drone; a step of creating a 3D model of the subject by aligning the plurality of scan data and identifying a plurality of meshes constituting the 3D model; a step of identifying a shooting blind spot of the drone for the subject among the plurality of meshes using a virtual camera in a pre-set coordinate space; and a step of setting an additional shooting path of the drone for the blind spot based on the normal vector of the mesh constituting the identified blind spot.

[0008] Additionally, the step of identifying the shooting blind spots includes the step of identifying candidate shooting blind spots by projecting rays onto the plurality of meshes using a virtual camera in a pre-set coordinate space.

[0009] Additionally, the step of selecting and identifying the shooting blind spot includes the step of selecting the shooting blind spot among the candidate shooting blind spots based on at least one of the area, shape, and point cloud density of the identified candidate shooting blind spot.

[0010] In addition, the method includes the step of identifying a plurality of meshes in the shape of triangles based on the point cloud of the three-dimensional model, identifying an edge where a mesh is not formed because the connection between adjacent triangles among the identified plurality of meshes is disconnected, and verifying candidate shooting blind spots based on the identified edge.

[0011] Additionally, the step of setting an additional shooting path of the drone includes the step of determining a plurality of candidate shooting locations located at a predetermined distance from the identified blind spot based on the normal vector of the mesh constituting the identified blind spot, and the step of determining an effective shooting location for the identified blind spot among the plurality of candidate shooting locations.

[0012] Additionally, the step of determining the effective shooting location includes determining the effective shooting location based on at least one of the distance between the candidate shooting location and the blind spot, whether the candidate shooting location exists in an open space, and whether the drone has a clear view from the candidate shooting location, and setting an additional shooting path of the drone based on the effective shooting location.

[0013] In addition, the step of setting an additional shooting path of the drone based on the above effective shooting location involves determining the shooting location of the multiple blind spots as the identified effective shooting location when an effective shooting location including multiple blind spots is identified among multiple effective shooting locations based on the drone's field of view.

[0014] Additionally, the step of setting an additional shooting path for the drone based on the valid shooting locations includes clustering the identified valid shooting locations based on the spatial distribution of the identified valid shooting locations, assigning each cluster to a plurality of drones, and setting the additional shooting path so that each drone circulates through the valid shooting locations within the assigned cluster.

[0015] To solve the above-mentioned problem, an electronic device for identifying a shooting blind spot and generating an additional shooting path based on mesh surface analysis of a subject identified using a drone according to one embodiment of the present disclosure includes a memory storing one or more instructions, a communication interface, and one or more processors that, by executing said instructions, acquire a plurality of point cloud scan data corresponding to said subject obtained by shooting the subject through the drone via said communication interface, align said plurality of scan data to generate a 3D model of said subject, identify a plurality of meshes constituting said 3D model, identify a shooting blind spot of said drone for said subject among said plurality of meshes using a virtual camera in a pre-set coordinate space, and set an additional shooting path of said drone for said blind spot based on the normal vector of the mesh constituting said blind spot.

[0016] In order to solve the above-mentioned problem, a computer program stored on a computer-readable storage medium according to one embodiment of the present disclosure, wherein the computer program, when executed on one or more processors, performs a method for identifying a shooting blind spot and generating an additional shooting path based on mesh surface analysis of a subject identified using a drone, and the method comprises the steps of: acquiring a plurality of point cloud scan data corresponding to the subject obtained by shooting the subject through a drone; aligning the plurality of scan data to create a three-dimensional model of the subject and identifying a plurality of meshes constituting the three-dimensional model; identifying a shooting blind spot of the drone for the subject among the plurality of meshes using a virtual camera in a pre-set coordinate space; and setting an additional shooting path of the drone for the blind spot based on the normal vector of the mesh constituting the identified blind spot. Effects of the invention

[0017] According to one embodiment of the present disclosure, blind spots in the shooting process that occur during 3D shooting using a drone can be efficiently recognized and compensated for. Accordingly, a 3D model of the external shape of the subject can be generated more precisely, and model distortion or alignment errors caused by incomplete data in the past can be effectively reduced.

[0018] In addition, by automatically identifying shooting blind spots and efficiently setting supplementary shooting positions, unnecessary procedures requiring operators to manually adjust flight paths or fly repeatedly can be reduced. Consequently, shooting efficiency is improved, and working time and energy consumption can be reduced.

[0019] In addition, multiple blind spots can be captured by using multiple drones. As a result, high-resolution 3D data can be obtained, and precision and reliability can be improved in application fields such as facility inspection, structural maintenance, and cultural property restoration. Brief explanation of the drawing

[0020] FIG. 1 is a drawing for explaining the operation of setting shooting points of each drone based on a virtual hemisphere surrounding a subject, using an electronic device according to one embodiment of the present disclosure. FIG. 2 is a block diagram for explaining the configuration of an electronic device according to one embodiment of the present disclosure. FIG. 3 is a flowchart illustrating the operation of an electronic device for setting sequential shooting points of each drone according to one embodiment of the present disclosure. FIG. 4 is a drawing for explaining a plurality of circular trajectories composed of individual drone shooting points set according to one embodiment of the present disclosure. FIG. 5 is a drawing for explaining an example of the wake range to which the wake influence of a drone according to one embodiment of the present disclosure is applied. FIGS. 6a and 6b are drawings illustrating a real-time control process of individual drones to avoid wake interference between drones performed by an electronic device according to one embodiment of the present disclosure. FIG. 7 is a flowchart of a method for identifying shooting blind spots and generating additional shooting paths based on mesh surface analysis of an identified subject using a drone according to one embodiment of the present disclosure. FIG. 8 is an example diagram of generating a three-dimensional model of a subject by aligning a plurality of scan data according to one embodiment of the present disclosure. FIG. 9 is an example diagram of identifying a blind spot for shooting a subject (1) using a virtual camera according to one embodiment of the present disclosure. FIG. 10a is an example diagram showing a conventional three-dimensional model, and FIG. 10b is an example diagram showing a three-dimensional model with a restored shooting blind spot according to one embodiment of the present disclosure. FIG. 11 is a table showing an index indicating the similarity of a three-dimensional model that has restored a shooting blind spot according to one embodiment of the present disclosure. Specific details for implementing the invention

[0021] The embodiments described herein are subject to various modifications and may have various forms; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the scope of specific embodiments and should be understood to include various modifications, equivalents, and / or alternatives of the embodiments of the present disclosure. In relation to the description of the drawings, similar reference numerals may be used for similar components.

[0022] In describing the present disclosure, if it is determined that a detailed description of related known functions or configurations could unnecessarily obscure the essence of the present disclosure, such detailed description is omitted.

[0023] Additionally, the following embodiments may be modified in various other forms, and the scope of the technical concept of the present disclosure is not limited to the following embodiments. Rather, these embodiments are provided to make the present disclosure more faithful and complete and to fully convey the technical concept of the present disclosure to those skilled in the art.

[0024] The terms used in this disclosure are used merely to describe specific embodiments and are not intended to limit the scope of the rights. The singular expression includes the plural expression unless the context clearly indicates otherwise.

[0025] In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of such features (e.g., numerical values, functions, actions, or components such as parts) and do not exclude the presence of additional features.

[0026] In the present disclosure, expressions such as “A or B,” “at least one of A or / and B,” or “one or more of A or / and B” may include all possible combinations of items listed together. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” may refer to cases including (1) at least one A, (2) at least one B, or (3) both at least one A and at least one B.

[0027] Expressions such as "first," "second," "first," or "second" used in this disclosure may modify various components regardless of order and / or importance, and are used only to distinguish one component from another and do not limit said components.

[0028] Where it is stated that a component (e.g., Component 1) is "(operatively or communicatively) coupled with / to" or "connected to" another component (e.g., Component 2), it should be understood that the component may be directly connected to the other component or connected through the other component (e.g., Component 3).

[0029] On the other hand, when it is stated that a certain component (e.g., a first component) is "directly connected" or "directly coupled" to another component (e.g., a second component), it may be understood that no other component (e.g., a third component) exists between the certain component and the other component.

[0030] The expression “configured to” as used in this disclosure may be replaced, depending on the context, with, for example, “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of.” The term “configured to” may not necessarily mean only “specifically designed to” in hardware.

[0031] Instead, in some situations, the expression “device configured to do something” may mean that the device is “capable of doing something” together with other devices or components. For example, the phrase “processor configured (or set) to perform A, B, and C” may mean a dedicated processor for performing those operations (e.g., an embedded processor), or a generic-purpose processor (e.g., a CPU or application processor) capable of performing those operations by executing one or more software programs stored in a memory device.

[0032] In the embodiments, a 'module' or 'part' performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Additionally, a plurality of 'modules' or a plurality of 'parts' may be integrated into at least one module and implemented by at least one processor, except for the 'module' or 'part' that needs to be implemented in specific hardware.

[0033] Meanwhile, various elements and areas in the drawings are depicted schematically. Accordingly, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0034] Hereinafter, embodiments according to the present disclosure are described in detail with reference to the attached drawings so that those skilled in the art can easily implement them.

[0035] FIG. 1 is a drawing for explaining the operation of setting shooting points of each drone based on a virtual hemisphere surrounding a subject, using an electronic device according to one embodiment of the present disclosure.

[0036] Referring to FIG. 1, an electronic device according to the present disclosure can identify a virtual hemispherical space (11) containing a subject (1). More specifically, the electronic device can identify a virtual hemispherical space (11) defined outside the subject (1) to enclose the subject (1) in a pre-set coordinate space (e.g., a global coordinate system, etc.).

[0037] At this time, when multiple drones each move along the outer edge of the hemispherical space (11) in a circular trajectory parallel to the floor surface, the overlap ratio between the images captured by each individual drone becomes constant, making precise restoration possible during 3D modeling. In particular, each drone can move along a different circular trajectory, and the circular trajectory can be set at a pre-set interval.

[0038] FIG. 2 is a block diagram for explaining the configuration of an electronic device according to one embodiment of the present disclosure.

[0039] The electronic device (100) may be implemented as a server (e.g., a control server) composed of at least one computer, or as a terminal device such as a remote control device, a smartphone, or a tablet PC. Additionally, it is possible for the electronic device (100) to be implemented as one of a plurality of drones that photograph a subject (1) (drone itself).

[0040] Referring to FIG. 2, the electronic device (100) may include a memory (110), a communication interface (120), and a processor (130).

[0041] The memory (110) is configured to store at least one instruction or data related to an operating system (OS) for controlling the overall operation of the components of the electronic device (100) and the components of the electronic device (100).

[0042] The memory (110) may include non-volatile memory such as ROM or flash memory, and may include volatile memory such as DRAM. Additionally, the memory (110) may include an auxiliary storage device such as a hard disk or SSD (Solid State Drive).

[0043] The communication interface (120) can be connected to various external network interfaces (e.g., drone, control server, etc.) through one or more networks, and can exchange data through various wired and wireless communication methods.

[0044] Depending on the area or scale, a network may be a Personal Area Network (PAN), Local Area Network (LAN), Wide Area Network (WAN), etc., and depending on the openness of the network, it may be an Intranet, Extranet, or Internet, etc.

[0045] Wireless communication may include at least one of the following communication methods: LTE (long-term evolution), LTE-A (LTE Advance), 5G (5th Generation) mobile communication, CDMA (code division multiple access), WCDMA (wideband CDMA), UMTS (universal mobile telecommunications system), WiBro (Wireless Broadband), GSM (Global System for Mobile Communications), DMA (Time Division Multiple Access), WiFi (Wi-Fi), WiFi Direct, Bluetooth, NFC (near field communication), Zigbee, etc.

[0046] Wired communication may include at least one of communication methods such as Ethernet, optical network, USB (Universal Serial Bus), Thunderbolt, HDMI, FireWire, and PLC (Power Line Communication).

[0047] The electronic device (100) can communicate with a plurality of drones (e.g., 10-1, 2, etc.), a control server, etc. through a communication interface (120). For example, the electronic device (100) can communicate directly with individual drones to receive information regarding the real-time location of each drone, shooting status (whether shooting is taking place, shooting direction, field of view, etc.), and the location of the subject (1), and can control the movement and shooting of each drone according to the trajectory of the individual drone derived according to the following embodiment. Alternatively, the electronic device (100) may receive information regarding the real-time location of the individual drone, shooting status, and the location of the subject (1) from the control server, and transmit information regarding the trajectory of the individual drone derived according to the following embodiment to the control server.

[0048] Meanwhile, communication methods are not limited to the examples described above and may include new communication methods that emerge with technological advancements.

[0049] The processor (130) is configured to control the overall configuration and operation of the electronic device (100).

[0050] The processor (130) is connected to the memory (110) and can control the electronic device (100) by executing at least one instruction stored in the memory (110).

[0051] To this end, the processor (130) may be implemented as a general-purpose processor such as a CPU (Central Processing Unit) or AP (Application Processor), a graphics-dedicated processor such as a GPU (Graphic Processing Unit) or VPU (Vision Processing Unit), or an artificial intelligence-dedicated processor such as an NPU (Neural Processing Unit). The processor (130) may include volatile memory such as SRAM.

[0052] Referring to FIG. 2, the processor (130) can control a trajectory setting module (131), a wake interference avoidance module (132), etc. These modules correspond to functional unit modules implemented in software and / or hardware.

[0053] The trajectory setting module (131) is configured to set a circular trajectory consisting of shooting points of individual drones according to the embodiments of FIGS. 1 and FIG. 3 described above, and ensures that the degree of overlap between images captured by each drone is constant.

[0054] The wake interference avoidance module (132) is a module for identifying the range of influence affected by the wake of each drone and controlling in real time so that other drones do not enter the range of influence.

[0055] FIG. 3 is a flowchart illustrating the operation of an electronic device for setting sequential shooting points of each drone according to an embodiment of the present disclosure. The operation of FIG. 3 can be performed by the trajectory setting module (131) described above.

[0056] Referring to FIG. 3, the electronic device (100) can create a virtual hemispherical space (11) that surrounds the subject (1) as described in FIG. 1 above (S310).

[0057] At this time, the hemispherical space (11) can be defined to have a minimum radius value that includes all of the subject (1).

[0058] And, the electronic device (100) can set a first shooting point of the first drone on the outer edge of the hemispherical space (S320). Referring to FIG. 2, the processor (130) can control a trajectory setting module (131), a wake interference avoidance module (132), etc. These modules correspond to functional unit modules implemented in software and / or hardware.

[0059] The trajectory setting module (131) is configured to set a circular trajectory consisting of shooting points of individual drones according to the embodiments of FIGS. 1 and FIG. 3 described above, and ensures that the degree of overlap between images captured by each drone is constant.

[0060] The wake interference avoidance module (132) is a module for identifying the range of influence affected by the wake of each drone and controlling in real time so that other drones do not enter the range of influence.

[0061] FIG. 3 is a flowchart illustrating the operation of an electronic device for setting sequential shooting points of each drone according to an embodiment of the present disclosure. The operation of FIG. 3 can be performed by the trajectory setting module (131) described above.

[0062] Referring to FIG. 3, the electronic device (100) can create a virtual hemispherical space (11) that surrounds the subject (1) as described in FIG. 1 above (S310).

[0063] At this time, the hemispherical space (11) can be defined to have a minimum radius value that includes all of the subject (1).

[0064] And, the electronic device (100) can set a first shooting point of the first drone on the outer edge of the hemispherical space (S320).

[0065] Specifically, the electronic device (100) can set the highest point on the outer edge of the hemispherical space (11) as the first shooting point of the first drone (10-1) located at a certain height (d) in the vertical direction relative to the highest point.

[0066] And, the electronic device (100) can set a circular trajectory consisting of a second shooting point of a second drone (10-2) on the outer edge of a hemispherical space (11) based on the coordinates of a first shooting point, the height of a first drone (10-1), and the field of view of each drone (assuming it is the same for all drones) (S330).

[0067] Specifically, an electronic device according to one embodiment of the present disclosure can determine a second shooting point by calculating the coordinates where the camera field of view of each drone meets the hemispherical surface, taking into account the field of view (θ) of the drone, the radius (r) of the subject (1), and the height (d) of the drone.

[0068] The straight line equation of the triangle according to the field of view range formed by the camera of the first drone (10-1) is as follows when defined as the field of view (θ):

[0069] (Form 1)

[0070] When determining the position of the second drone (10-2) that follows the first drone (10-1) by finding the angle β, the existence of a y value must first be checked, and this discriminant is as in Equation 2.

[0071] (Form 2)

[0072] Here In the case of, it means that a real root exists, and if there is no real root, it indicates that the drone's field of view is not inscribed with the subject (1). Angle When calculating, the y-value must be calculated, which is equal to mathematical formula 3.

[0073] (Form 3)

[0074] Now, based on the y value obtained, the β value required to determine the position of the next drone can be calculated.

[0075] Specifically, the electronic device (100) can calculate a target angle to define a circular trajectory to which a second shooting point belongs in relation to the center point of the hemispherical space and the first shooting point, based on the coordinates of the first shooting point, the height of the first drone (10-1) relative to the first shooting point, and the radius of the hemispherical space (11).

[0076] The target angle refers to the angle between the first shooting point and the second shooting point based on the center of the hemispherical space (11).

[0077] To calculate the target angle β value, an overlap value is required, which is defined as Equation 4. Here, the overlap is defined as p.

[0078] (Fourth form)

[0079] Additionally, the electronic device (100) can calculate a shooting interval on the outer edge of the hemispherical space (11) based on the aforementioned target angle, and based on the shooting interval, identify at least one circular trajectory to which the third shooting point of at least one third drone belongs on the outer edge of the hemispherical space (11).

[0080] Specifically, the electronic device (100) calculates the number of equal shooting points N and the distance L between shooting intervals when divided vertically using the calculated β value, and using this value, can calculate the distance between horizontal shooting points based on vertical shooting points using Equation 5.

[0081] (Form 5)

[0082] Through this process, the drones can move along the hemisphere while maintaining a constant overlap ratio and position shooting points, and the result is shown in Fig. 4.

[0083] Meanwhile, when multiple drones fly in the same space, wake interference between them can occur, potentially degrading flight stability. To address this, it is necessary to determine an optimal flight path that is unaffected by wake interference.

[0084] In this regard, an electronic device (100) according to one embodiment of the present disclosure controls the movement of each drone along a circular trajectory composed of shooting points of individual drones as described above, and can perform control to prevent wake interference between drones in real time.

[0085] In this regard, FIG. 5 is a drawing for explaining an example of the wake range affected by the wake influence of a drone according to one embodiment of the present disclosure.

[0086] For example, the electronic device (100) can identify the range of influence affected by the wake of the second drone (10-2) according to the real-time position of the second drone (10-2), and control the third drone (10-3) in real-time to photograph the subject (1) at a point that does not fall within the range of influence of the second drone (10-2) in the circular trajectory of the third drone (10-3). That is, as shown in FIG. 5, the third drone (10-3) is controlled so that a situation in which it falls within the range of influence of the second drone (10-2) does not occur.

[0087] In one embodiment, the electronic device (100) may define the angle affecting the wake as θ and the height as h. At this time, the radius of the base of the cone representing the wake region is defined as r'. The position of the upper drone (x t , y t , z t ), when another drone is located at (x0, y0, z0), the relative z-direction distance t = z0 - z t is 0 <t< h를 만족해야 하며, 해당 높이에서의 원뿔의 반지름은 수학식 6과 같다.

[0088] (6th form)

[0089] At the same time, the horizontal distance between the drone and the z-axis is equal to Equation 7.

[0090] (Seventh Form)

[0091] d≤r t In this case, the electronic device (100) determines that the drone is within the wake influence range. Accordingly, the electronic device (100) can adjust the circulating path of the sub-drone to avoid the wake influence.

[0092] In this regard, FIGS. 6a and 6b are drawings illustrating a real-time control process of individual drones to avoid wake interference between drones, performed by an electronic device according to an embodiment of the present disclosure. In each of the screens of FIGS. 6a and 6b, the upper UI indicates the real-time position of an individual drone (node) when viewed from above in a hemispherical space (11), and the lower UI indicates the horizontal angle of the individual drone over time (an angle in the horizontal direction relative to the center point of the hemispherical space). A situation in which other drones located below each drone are controlled so as not to be located on a horizontal angle corresponding to the wake range in real time is confirmed according to the graph.

[0093] Compared to FIG. 6a, FIG. 6b shows that a larger proportion of the circular trajectory of each drone is filled, which can be understood as a situation where the shooting progress of each drone is filled as shooting proceeds in real time. That is, shooting ends when each drone completes one full rotation to capture a shooting point according to its respective set circular trajectory. Here, each drone is able to complete the rotation while avoiding the influence of wake interference from other drones by means of the wake interference avoidance module (132) described above.

[0094] Hereinafter, an embodiment of the present disclosure is described for detecting a blind spot in the shooting of a subject (1) based on scan data of the subject (1) obtained through a drone.

[0095] FIG. 7 is a flowchart of a method for identifying shooting blind spots and generating additional shooting paths based on mesh surface analysis of a subject (1) identified using a drone according to one embodiment of the present disclosure.

[0096] Referring to FIG. 7, the processor (130) can obtain a plurality of point cloud scan data corresponding to the acquired subject (1) by photographing the subject (1) through a drone (S710).

[0097] Specifically, the point cloud scan data may be a set of data including three-dimensional coordinate values ​​(x, y, z) in a pre-set coordinate space (e.g., a global coordinate system, etc.) for a plurality of points constituting the surface of the subject (1), obtained by scanning the surface of the subject (1) by a drone (10-1 to 10-3, hereinafter 10). For example, the point cloud scan data may be a set of data including at least one of intensity, color (RGB), timestamp, normal vector, and confidence along with location information (i.e., coordinate value) for each point.

[0098] Point cloud scan data can be obtained using the LiDAR, structured light, stereo camera, or photogrammetry method of the drone (10).

[0099] The processor (130) can obtain multiple point cloud scan data corresponding to the subject (1) from at least one drone (10) that captured the subject (1) through a communication interface.

[0100] In particular, the processor (130) can receive point cloud scan data from a plurality of drones (10), and each drone (10) can collect sensor data according to set field of view, shooting distance, and resolution conditions while sequentially or in parallel following a plurality of shooting points in a virtual space (e.g., a hemispherical or spherical outer edge) defined around the subject (1). The plurality of drones (10) configured according to the above-described embodiment can transmit point cloud scan data acquired at each shooting point to the processor (130) in real-time or asynchronously, and the processor (130) can integrate this to form a plurality of point cloud data representing the outer shape of the subject (1).

[0101] Meanwhile, each drone (10) can acquire attitude and position information of the drone (10) using GNSS and IMU sensors and transmit it to a processor (130), including metadata such as internal parameters of the sensor (e.g., focal length, principal point, distortion coefficient) and external parameters (e.g., coordinate transformation relationship between the sensor and the aircraft) and the time of shooting. The processor (130) can use the metadata to improve the precision of subsequent registration and coordinate transformation processes.

[0102] Additionally, the processor (130) controls communication between multiple drones (10) to minimize overlap in the shooting area and can dynamically adjust the flight path by considering the remaining battery, altitude, and field of view of each drone (10). The processor (130) can store multiple point cloud scan data acquired according to the control, separated by drone (10) or by frame, and input them for alignment and integration processing in a subsequent step.

[0103] FIG. 8 is an example diagram of generating a three-dimensional model of a subject (1) by aligning a plurality of scan data according to one embodiment of the present disclosure.

[0104] The processor (130) can acquire multiple scan data, align the multiple scan data to create a three-dimensional model of the subject (1), and identify multiple meshes that constitute the three-dimensional model (S720).

[0105] Specifically, the processor (130) can perform a preprocessing process to remove noise and unnecessary planar regions within each scan data prior to alignment. The processor (130) can improve the stability of subsequent alignment by applying a RANSAC (Random Sample Consensus) based planar estimation algorithm to detect floor surfaces or background planes unrelated to the subject (1) and removing points included in the area.

[0106] The processor (130) can perform initial matching between adjacent scans using Fast Point Feature Histogram (FFFH) features for each preprocessed scan data. The FPFH algorithm has the characteristic of being able to efficiently calculate local structural similarity between points by statistically summarizing the normal vector relationships and spatial geometric characteristics around each point and expressing them as feature descriptors. By performing initial matching between adjacent scans using these FPFH features, the processor (130) can enable stable correspondence estimation even for objects (1) with complex structures.

[0107] The processor (130) can apply the Random Sample Consensus (RANSAC) algorithm based on the initial registration results to remove outlier corresponding points and improve the reliability of the registration transformation matrix. The processor (130) can then perform fine registration between adjacent scans by performing the Iterative Closest Point (ICP) algorithm using the initial registration results refined through the RANSAC algorithm. The ICP algorithm can improve registration accuracy by progressively correcting the relative transformation (rotation and translation) between two scans through iterative optimization based on the nearest distance between adjacent points.

[0108] Additionally, the processor (130) may perform pose graph optimization to minimize the cumulative error between multiple scan data. Pose graph optimization can be performed by defining the position and pose of each scan as nodes and the relative alignment relationship between scans as edges to minimize the global error function, thereby enabling the processor (130) to achieve consistent global alignment of the entire scan set.

[0109] Referring to FIG. 8, the processor (130) can remove misaligned points and residual noise based on the alignment and optimization results described above in a pre-set coordinate space (600) and generate an integrated point cloud that reflects the overall shape of the subject (1). At this time, the integrated point cloud is three-dimensional point cloud data composed of multiple points constituting the surface of the subject (1), and can serve as basic data for a three-dimensional model that spatially represents the outer shape and detailed structure of the subject (1).

[0110] The processor (130) can generate multiple meshes that approximate the surface of the subject (1) by applying an alpha-shape (α-shape)-based surface reconstruction algorithm to an integrated point cloud. The alpha-shape algorithm is based on the spatial distribution between points, and the α parameter

[0111] This is a technique for generating a polyhedral shape that encloses the outer edge of a point cloud according to a value. By adjusting the α parameter, the processor (130) can control the resolution and continuity of the mesh, such as preserving the fine outer shape of the subject (1) or removing unnecessary fine irregularities. Accordingly, referring again to FIG. 8, the processor (130) can efficiently restore an outer surface similar to the actual shape of the subject (1), and the generated multiple meshes can be used as reference data for analyzing shooting blind spots and determining visibility in subsequent steps.

[0112] FIG. 9 is an example diagram of identifying a blind spot for shooting a subject (1) using a virtual camera according to one embodiment of the present disclosure.

[0113] And, the processor (130) can identify a blind spot for shooting of a subject (1) among a plurality of meshes using a virtual camera in a pre-set coordinate space (600) (730). The blind spot for shooting may be an area where the drone (10) camera's field of view does not reach or point cloud data is not sufficiently acquired due to the complex shape of the subject (1), shading, or limitations on the shooting angle.

[0114] Specifically, the processor (130) may define a virtual camera within a pre-set coordinate space (600) (e.g., a global coordinate system, etc.). The virtual camera may be configured to simulate the shooting position and field of view conditions of the actual drone (10), and may be positioned using, for example, position, attitude (roll, pitch, yaw), and field of view information calculated from the shooting trajectory or flight log data of the drone (10). Through this virtual camera setup, the processor (130) can identify shooting blind spots including a visible area and a non-visible area in the actual shooting environment.

[0115] According to one embodiment of the present disclosure, the processor (130) can identify candidate shooting blind spots by projecting rays onto a plurality of meshes using a virtual camera in a preset coordinate space (600).

[0116] Specifically, the processor (130) can ray-cast multiple rays onto multiple meshes from a virtual camera and calculate whether there is an intersection between the direction of travel of each ray and the normal vector of the mesh surface. The processor (130) can determine the surface where each ray first intersects as an observable outer surface and determine a surface located further back on the same ray path that the ray cannot reach as a candidate shooting blind spot. That is, if a ray is blocked by another mesh and cannot reach a specific surface, the processor (130) considers that surface as an area that is unobservable from the virtual camera viewpoint. Referring to FIG. 9, using the results of such ray-mesh intersection calculations, the processor (130) can identify the surface of a subject (1) that is likely to have point cloud data missing during actual drone (10) shooting as a candidate shooting blind spot.

[0117] At this time, the processor (130) can perform filtering using the ray casting results to leave only the surfaces that are visible from the outside. Specifically, the processor (130) can project rays in multiple directions from a virtual camera, determine and maintain the mesh surface where each ray first intersects as the outer surface, and remove the inner mesh or surfaces corresponding to the inner cavity located behind it to identify candidate shooting blind spots. Through this, the processor (130) can prevent abnormal areas corresponding to empty spaces or mesh holes in internal structures from being incorrectly identified as shooting blind spots, and can leave only the supplementary shooting areas that require actual external observation.

[0118] In addition, according to one embodiment of the present disclosure, the processor (130) can select a shooting blind spot among the candidate shooting blind spots based on at least one of the area, shape, and density of the point cloud of the identified candidate shooting blind spot.

[0119] Additionally, the processor (130) can select actual shooting blind spots among the identified candidate shooting blind spots based on at least one criterion among area, shape, and point cloud density. For example, if the area of ​​the region corresponding to each candidate shooting blind spot is greater than or equal to a preset first threshold, the processor (130) may exclude the region by determining that the region is likely the entire outer edge of the subject (1). Conversely, if the area is less than a second threshold, it may exclude the region by determining it as a fine noise region. Additionally, if the density of the point cloud is less than or equal to a preset reference value, it may be determined as a shooting blind spot. Along with this, the processor (130) may analyze the shape characteristics of the region corresponding to each candidate shooting blind spot, such as convexity and aspect ratio, to distinguish between actual structural empty space (i.e., an area excluded from the drone's (10) field of view) and abnormal boundaries caused by simple mesh defects or noise. Through this, the processor (130) can exclude false positives caused by noise based on the spatial clustering results of the point cloud and accurately determine only the area excluded from the actual drone (10) field of view.

[0120] Meanwhile, the processor (130) can identify a plurality of meshes in the shape of triangles based on the point cloud of a three-dimensional model, identify an edge where the connection between adjacent triangles among the identified meshes is disconnected and no mesh is formed, and verify candidate shooting blind spots based on the identified edge.

[0121] The processor (130) can analyze the connection structure between multiple triangle meshes identified from the point cloud of a three-dimensional model to identify an edge where the connection between adjacent triangles is disconnected and no mesh is formed. The edge may be a section of shape change of the actual subject (1) or a section formed due to a missed shot, and the processor (130) can verify the location and validity of candidate shooting blind spots based on the edge. Through this, the processor (130) can identify not only simple shaded areas but also blind spots caused by structural disconnections of the actual subject (1).

[0122] According to one embodiment of the present disclosure, the processor (130) can set an additional shooting path of the drone (10) for the blind spot based on the normal vector of the mesh constituting the identified blind spot (S740).

[0123] The processor (130) can calculate a normal vector corresponding to each triangular face of the mesh constituting the identified shooting blind spot and average the normal vectors of a plurality of adjacent triangular faces to produce a representative normal vector of the blind spot. In particular, since a normal vector of a single point may be sensitive to noise, the processor (130) can derive a direction vector by summing and normalizing k normal vectors (k is a natural number greater than or equal to 2) calculated from a plurality of meshes constituting the shooting blind spot according to the 8th equation.

[0124] (Form 8)

[0125]

[0126] The representative normal vector calculated in this way can be used as a reference direction for generating the shooting direction or shooting position of the shooting blind spot. Additionally, the processor (130) can set an additional shooting path for the shooting blind spot based on the identified shooting direction and shooting position.

[0127] The processor (130) can determine a plurality of candidate shooting positions located at a predetermined distance from the identified blind spot based on the normal vector of the mesh constituting the identified blind spot, and determine an effective shooting position for the identified blind spot among the plurality of candidate shooting positions.

[0128] The processor (130) can determine multiple candidate shooting positions by calculating positions corresponding to multiple distance candidates (e.g., 0.8m, 1.0m, 1.5m, 2.0m) from the center point of the blind spot along a representative normal vector. The processor (130) can calculate each candidate shooting position according to Equation 9 below.

[0129] (Form 9)

[0130] Here is the center point of the blind spot, is a distance scalar value, may be an average normal vector. The processor (130) can determine the shooting direction at each candidate shooting location by associating the opposite direction of the normal vector with each candidate shooting location.

[0131] The processor (130) determines an effective shooting location based on at least one of the distance between a candidate shooting location and a blind spot, whether the candidate shooting location exists in an open space, and whether the drone (10) has a line of sight at the candidate shooting location, and can set an additional shooting path for the drone (10) based on the effective shooting location. Specifically, the processor (130) can determine an effective shooting location for a plurality of candidate shooting locations based on at least one of the following conditions. For example, the processor (130) can avoid the risk of collision by determining whether the distance between the candidate shooting location and the blind spot surface is greater than or equal to a set minimum safety distance (e.g., 0.5 m). In addition, the processor (130) can exclude the area inside or below the structure by determining whether the candidate shooting location exists in an open space rather than inside the mesh. In addition, the processor (130) can determine whether a line of sight is secured by analyzing whether there is an obstacle on the straight path from the candidate location to the target point in the blind spot. The processor (130) can determine a candidate shooting location that satisfies all of the above-described conditions as a valid shooting location and generate a set of valid shooting locations including these locations.

[0132] When a processor (130) identifies a valid shooting location that includes multiple blind spots among multiple valid shooting locations based on the field of view of the drone (10), the processor (130) can determine the shooting location of the multiple blind spots as the identified valid shooting location. Specifically, for a set of generated valid shooting locations, the processor (130) can calculate the area of ​​the blind spots that can be shot from each valid shooting location based on the camera field of view (FOV) of the drone (10). The processor (130) can identify a valid shooting location that simultaneously includes two or more blind spots among multiple valid shooting locations, and minimize the number of shooting locations by applying the identified valid shooting location to the multiple blind spots in common. Through this, the processor (130) can determine an optimal valid shooting location that can compensate for the entire blind spot with a minimum number of additional shots.

[0133] Additionally, the processor (130) may cluster the identified valid shooting locations based on the spatial distribution of the identified valid shooting locations, assign each cluster to a plurality of drones (10), and set additional shooting paths so that each drone (10) circulates the valid shooting locations within the assigned cluster.

[0134] Specifically, the processor (130) can cluster effective shooting locations based on the spatial distribution of a plurality of determined effective shooting locations. The processor (130) can group adjacent effective shooting locations by applying a K-Means clustering algorithm and assign each cluster to one of a plurality of drones (10). Additionally, the processor (130) can calculate a path for each drone (10) to traverse multiple effective shooting locations within the assigned cluster and set the path using a Traveling Salesman Problem (TSP)-based optimization algorithm so that the sum of the total flight distances is minimized. The processor (130) can control the path to prevent collisions and ensure that parallel additional shooting missions are performed safely and efficiently by adjusting the path while considering minimum distance constraints between drones (10) (e.g., 2m or more).

[0135] Meanwhile, the processor (130) can transmit additional path information set through the communication interface (120) to the drone (10).

[0136] FIG. 10a is an example diagram showing a conventional three-dimensional model, and FIG. 10b is an example diagram showing a three-dimensional model with a restored shooting blind spot according to one embodiment of the present disclosure.

[0137] Referring to FIG. 10a, in conventional fixed path-based shooting, there was a problem where a shooting blind spot occurred in the lower area of ​​a complex shape, such as the eaves of a building (300). For example, in the conventional shooting result shown on the left side of FIG. 10a, a missing area is identified because some of the lower surfaces of the building (300) are not reconstructed into mesh data. On the other hand, as a result of applying the mesh-based blind spot detection and multi-drone (10) reinforcement shooting technique according to one embodiment of the present disclosure, it can be visually confirmed that the missing area was successfully supplemented, as shown in FIG. 10b.

[0138] FIG. 11 is a table showing an index indicating the similarity of a three-dimensional model that has restored a shooting blind spot according to one embodiment of the present disclosure.

[0139] Figure 11 quantitatively compares the restoration results of Figure 10 and shows the results of comparing the quality of a 3D model after applying the conventional fixed path shooting method and the reinforcement shooting method of the present disclosure. The SSIM (Structural Similarity Index, ↑) indicator, which indicates the structural similarity of the model, improved from 0.8587 of the conventional method to 0.8809 of the proposed method, and the LPIPS (Learned Perceptual Image Patch Similarity, ↓) indicator, which indicates the loss of perceptual quality, decreased from 0.2835 to 0.1875. These results demonstrate that the reinforcement shooting method of the present disclosure can significantly improve the precision and completeness of the 3D model compared to the conventional shooting method. Therefore, through the mesh-based blind spot additional shooting method according to one embodiment of the present disclosure, the problem of missing shots occurring in the lower part or shaded area of ​​a complex structure can be resolved, and consequently, the effect of qualitatively and quantitatively improving the quality of the 3D modeling can be achieved.

[0140] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0141] Although preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above. It is understood that various modifications can be made by those skilled in the art without departing from the essence of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical spirit or perspective of the present disclosure. Explanation of the symbols

[0142] 100: Electronic device 110: Memory 120: Communication interface 130: Processor

Claims

Claim 1 A method for identifying shooting blind spots and generating additional shooting paths based on mesh surface analysis of a subject identified using a drone, comprising: a step of acquiring a plurality of point cloud scan data corresponding to the subject obtained by photographing the subject through a drone; a step of aligning the plurality of scan data to generate a 3D model of the subject and identifying a plurality of meshes constituting the 3D model; and a step of identifying a shooting blind spot of the drone for the subject among the plurality of meshes using a virtual camera in a pre-set coordinate space. The method includes the step of setting an additional shooting path for the drone for the blind spot based on the normal vector of the mesh constituting the identified blind spot; wherein the step of identifying the shooting blind spot comprises: the step of identifying candidate shooting blind spots by projecting rays onto the plurality of meshes using the virtual camera in the pre-set coordinate space; the step of projecting rays in multiple directions from the virtual camera, maintaining the mesh surface where each ray first intersects as the outer surface, and removing from the candidate shooting blind spot the mesh surface corresponding to the inner mesh or inner cavity located behind the outer surface; and the step of identifying an edge where the connection between adjacent triangles among the plurality of meshes is severed and no mesh is formed, and verifying the candidate shooting blind spot based on the identified edge. A method comprising the step of selecting the shooting blind spots by excluding, based on the area of ​​the identified candidate shooting blind spots, candidate shooting blind spots whose area is greater than or equal to a preset first threshold and candidate shooting blind spots whose area is less than a preset second threshold. Claim 2 delete Claim 3 delete Claim 4 delete Claim 5 A method according to claim 1, wherein the step of setting an additional shooting path of the drone comprises: determining a plurality of candidate shooting positions located at a predetermined distance from the identified blind spot based on a normal vector of a mesh constituting the identified blind spot; and determining an effective shooting position for the identified blind spot among the plurality of candidate shooting positions. Claim 6 In claim 5, the step of determining the effective shooting location comprises: determining the effective shooting location based on at least one of the distance between the candidate shooting location and the blind spot, whether the candidate shooting location exists in an open space, and whether the drone's line of sight is secured at the candidate shooting location; and setting an additional shooting path of the drone based on the effective shooting location. Claim 7 In claim 6, the step of setting an additional shooting path of the drone based on the effective shooting location is a method in which, if an effective shooting location including a plurality of blind spots among a plurality of effective shooting locations is identified based on the field of view of the drone, the shooting location of the plurality of blind spots is determined as the identified effective shooting location. Claim 8 In claim 7, the step of setting an additional shooting path of the drone based on the valid shooting location comprises: clustering the identified valid shooting locations based on the spatial distribution of the identified valid shooting locations, assigning each cluster to a plurality of drones, and setting the additional shooting path such that each drone circulates the valid shooting locations within the assigned cluster; the method. Claim 9 An electronic device for identifying shooting blind spots and generating additional shooting paths based on mesh surface analysis of a subject identified using a drone, comprising: a memory storing one or more instructions; and a communication interface. And by executing the above instructions, one or more processors include: acquiring a plurality of point cloud scan data corresponding to the subject obtained by photographing the subject via a drone through the communication interface; aligning the plurality of scan data to generate a 3D model of the subject; identifying a plurality of meshes constituting the 3D model; identifying a blind spot for the drone's shooting of the subject among the plurality of meshes using a virtual camera in a preset coordinate space; and setting an additional shooting path for the drone for the blind spot based on the normal vector of the mesh constituting the identified blind spot. The one or more processors include: projecting rays onto the plurality of meshes using the virtual camera in the preset coordinate space to identify candidate shooting blind spots; projecting rays in multiple directions from the virtual camera to maintain the mesh surface where each ray first intersects as the outer surface; removing a mesh surface corresponding to an inner mesh or inner cavity located behind the outer surface from the candidate shooting blind spot; and a mesh where the connection between adjacent triangles among the plurality of meshes is severed and no mesh is formed. An electronic device that identifies an edge, verifies the candidate shooting blind spot based on the identified edge, and selects the shooting blind spot by excluding, based on the area of ​​the identified candidate shooting blind spot, the candidate shooting blind spot whose area is greater than or equal to a preset first threshold and the candidate shooting blind spot whose area is less than a preset second threshold. Claim 10 A computer program stored on a computer-readable storage medium, wherein the computer program, when executed on one or more processors, performs a method for identifying shooting blind spots and generating additional shooting paths based on mesh surface analysis of a subject identified using a drone, the method comprising: a step of acquiring a plurality of point cloud scan data corresponding to the subject obtained by photographing the subject through a drone; a step of aligning the plurality of scan data to generate a 3D model of the subject and identifying a plurality of meshes constituting the 3D model; and a step of identifying a shooting blind spot of the drone for the subject among the plurality of meshes using a virtual camera in a pre-set coordinate space. The method includes the step of setting an additional shooting path for the drone for the blind spot based on the normal vector of the mesh constituting the identified blind spot; wherein the step of identifying the shooting blind spot comprises: a step of identifying candidate shooting blind spots by projecting rays onto the plurality of meshes using the virtual camera in the pre-set coordinate space; a step of projecting rays in multiple directions from the virtual camera, maintaining the mesh surface where each ray first intersects as the outer surface, and removing from the candidate shooting blind spot the mesh surface corresponding to the inner mesh or inner cavity located behind the outer surface; and a step of identifying an edge where the connection between adjacent triangles among the plurality of meshes is severed and no mesh is formed, and verifying the candidate shooting blind spot based on the identified edge. A computer program comprising: a step of selecting the shooting blind spots by excluding, based on the area of ​​the identified candidate shooting blind spots, candidate shooting blind spots whose area is greater than or equal to a preset first threshold and candidate shooting blind spots whose area is less than a preset second threshold.

Images