Omnidirectional image acquisition device and method for omnidirectional image acquisition

Abstract

The application provides a sensing and computing integrated chip, an unmanned aerial vehicle line laser three-dimensional reconstruction method and system. The chip comprises a processing array, the processing array is composed of a two-dimensional array of a plurality of processing units, each processing unit corresponds to a pixel in an input image, and is used for performing a morphological skeleton extraction operation through local parallel processing to identify a center path of a target in the image and output a two-dimensional center coordinate sequence of the target. The application realizes deep fusion of sensing and computing by one-to-one correspondence between the processing array and the image sensor pixels and local parallel execution of the morphological skeleton extraction operation, significantly shortens the data processing path, enables the laser line center extraction process to be completed within ten microseconds, effectively supports high-frequency acquisition and real-time response, and breaks through the delay bottleneck of the traditional sensing, transmission and computing architecture.

Description

Technical Field

[0001] This invention relates to the field of 3D vision technology, and in particular to an integrated sensing, storage, and computing chip, a method and system for line laser 3D reconstruction of unmanned aerial vehicles. Background Technology

[0002] In the fields of industrial automation and digital surveying, 3D reconstruction technology has become an important means of acquiring high-precision geometric information of the physical world. Currently, the mainstream technologies are mainly divided into two types: fixed active scanning systems and mobile passive imaging systems.

[0003] Fixed active scanning systems, especially line laser-based scanners, excel in industrial quality control, such as in conveyor belt inspection applications. These systems achieve sub-millimeter measurement accuracy by moving the object to be measured at a precisely controlled speed through a fixed sensor. The active light source (laser) makes them unaffected by ambient light and surface texture, providing reliable measurements on a wide range of materials. However, a key limitation of these systems lies in their "static" nature: they require the object to be measured to be brought close to the scanner, which significantly restricts their applications, preventing them from scanning large, irregular, or immovable objects such as bridges, dams, buildings, or extensive terrain.

[0004] In contrast, mobile passive imaging technologies, such as photogrammetry using cameras mounted on unmanned aerial vehicles (UAVs), especially the application of Structure from Motion (SfM) algorithms, have greatly promoted the development of large-scale 3D mapping. UAV photogrammetry technology, with its high mobility, low cost, and scalability, has been widely used in fields such as construction, agriculture, and disaster monitoring. Traditional airborne vision solutions follow the classic von Neumann architecture of "perception-transmission-computation." Under this architecture, the airborne camera is responsible for capturing massive amounts of 2D image data. This data (typically several megabytes per frame) must be transmitted completely via a data bus to a separate, high-performance airborne accompanying computer or ground station for processing. This process has the following main drawbacks: Real-time bottlenecks and high computational latency: 3D reconstruction, especially photogrammetric workflows based on Structure from Motion (SfM) and Multi-View Stereo (MVS), is a computationally intensive task. The entire post-processing process typically takes hours or even days to complete, which is completely unacceptable for applications such as emergency response and dynamic monitoring that require real-time acquisition of 3D information. Even with simplified real-time localization and mapping (SLAM) technology, the high computational demands make achieving high-precision, high-density real-time 3D reconstruction on resource-constrained UAV platforms extremely challenging.

[0005] In addition to these challenges, there are also power consumption bottlenecks and high demands on computing units. To process massive amounts of image data, the airborne system must be equipped with high-performance computing units (such as CPUs / GPUs), which themselves consume a significant amount of power. The high-speed data transfer between sensors, memory, and processors also generates substantial power consumption. For small drones that rely primarily on batteries for power, every watt of power consumption directly impacts their flight endurance.

[0006] A search revealed that patent 202210830487.0 discloses a method for extracting the three-dimensional coordinates of the center line of a line laser stripe, which extracts the center line and reconstructs the three-dimensional coordinates through image processing. Its drawback lies in its reliance on a general-purpose processor for serial computation, failing to achieve parallel real-time processing under an integrated sensing-memory-computing architecture. This results in high data transmission load and large processing latency, making it difficult to meet the low-latency, high-efficiency three-dimensional reconstruction requirements of high-speed UAV flight scenarios. Summary of the Invention

[0007] To address the shortcomings of existing technologies, the purpose of this application is to provide an integrated sensing-memory-computing chip, a method and system for UAV line laser 3D reconstruction, which can fundamentally break the traditional "sensing-transmission-computing" architecture and achieve high-precision 3D reconstruction while meeting the two major requirements of high real-time performance (high-speed processing) and low power consumption for 3D reconstruction of UAV platforms.

[0008] In a first aspect, this application provides a sensing-memory-computing integrated chip, including a processing array, wherein the processing array is composed of a two-dimensional array of multiple processing units, each processing unit corresponding to a pixel of an input image, sensing the original data of the pixel and storing it locally. Each processing unit accesses the data stored in itself and adjacent processing units to perform morphological skeleton extraction operations in parallel, identify the center path of the target in the image, and output the two-dimensional center coordinate sequence of the target.

[0009] Optionally, it also includes a main control unit, which is used to issue unified instructions to the processing array, schedule the parallel computing tasks of each processing unit, receive the processing results, and complete the aggregation, packaging and output of data.

[0010] Optionally, the processing unit includes: A pixel data caching module is used to receive and cache raw pixel data from the image sensor; An analog register is used to store the raw pixel data; The communication interface supports data interaction with adjacent processing units in the north, east, south, and west, enabling neighborhood access in non-von Neumann architectures. The arithmetic logic unit performs morphological skeleton extraction operations based on local analog register data and neighborhood data; Digital registers are used to store intermediate calculation results of arithmetic logic units; The status flag is used to indicate whether the processing unit has undergone a value update within the current processing cycle.

[0011] A second aspect of this application provides a method for line laser 3D reconstruction of unmanned aerial vehicles (UAVs), employing the aforementioned integrated sensing, storage, and computing chip, comprising: A laser beam is projected onto the target scene, forming laser stripes that change with the terrain; The laser stripe image is captured in real time, and morphological skeleton extraction is performed on the pixels of the laser stripe image in parallel to identify the center path of the target in the image, output the two-dimensional center coordinate sequence of the target, and simultaneously cache the corresponding grayscale image. Receive the two-dimensional center coordinate sequence and simultaneously record grayscale images, inertial measurement unit (IMU) data, and global navigation satellite system (GNSS) data; A georeferenced 3D point cloud model is generated based on multi-source synchronous data.

[0012] Optionally, the parallel processing of the pixels in the laser stripe image to perform morphological skeleton extraction, identify the center path of the target in the image, and output the two-dimensional center coordinate sequence of the target includes: The input laser stripe image is binarized to obtain a binary image of the laser stripes; Perform a parallel erosion operation on the binary image to remove isolated bright pixels while retaining the main stripe structure; In the iterative refinement stage, multiple sets of structural elements are used to sequentially perform hit-miss transformation on the main stripe structure to identify the current edge pixels. In each iteration, the identified edge points are synchronously set to zero to achieve the layer-by-layer peeling of the boundary of the main stripe structure. Repeat the above process until no processing unit state update occurs in a certain iteration, at which point the skeleton is considered to have converged. After the skeleton converges, the output is a sparse binary image containing only a single-pixel-width center line, and the corresponding two-dimensional center coordinate sequence is generated.

[0013] Optionally, the iterative refinement stage involves sequentially performing hit-and-miss transformations on multiple sets of structural elements to identify the current edge pixels, and synchronously setting the identified edge points to zero in each iteration to achieve layer-by-layer peeling of the boundaries of the main stripe structure, including: Enter the loop iteration process, and in each round, call multiple pre-set fixed structure elements in sequence; For each set of structure elements, all processing units perform the following operations in parallel: Centered on the corresponding pixel, the binarized states of the adjacent processing units in the north, east, south, and west are obtained through the communication interface to form 4-neighborhood data; By combining local pixel values ​​with neighborhood data, it is determined whether the pixel matches the edge pattern defined by the current structuring element. If it matches, the pixel is determined to be a removable edge point. All processing units that are identified as edge points synchronously set their output to zero, thereby achieving parallel edge trimming of the image boundary; After each round, each processing unit updates its local status flag based on whether its value has changed; When all structural elements have been fully traversed in a certain round, and the status flags of all processing units have not been set, the iteration terminates and the skeleton extraction is determined to be complete.

[0014] Optionally, the three-dimensional reconstruction based on the synchronized two-dimensional center coordinates data stream, image information, and orientation information includes: Based on inertial measurement unit (IMU) and global navigation satellite system (GNSS) data, combined with sparse feature matching of grayscale images, the six-degree-of-freedom pose of each frame image is calculated through joint optimization of post-differential positioning (PPK) and structure for motion recovery (SfM). Combining the camera calibration parameters, the six-degree-of-freedom pose, and the two-dimensional center coordinate sequence of the laser stripes, the two-dimensional coordinates of each frame are back-projected along the imaging optical path using the triangulation method, and the spatial intersection points with the pre-calibrated laser plane are solved to generate the local three-dimensional point cloud corresponding to that frame. Based on the timestamps and six-degree-of-freedom poses of each frame, all local 3D point clouds are registered to a unified global coordinate system through rigid body transformation, and then fused to generate a complete and geo-registered 3D point cloud model.

[0015] Optionally, the six-degree-of-freedom pose of each frame of the image is calculated by combining inertial measurement unit (IMU) and global navigation satellite system (GNSS) data with sparse feature matching of grayscale images, and through joint optimization of post-differential positioning (PPK) and structure for motion recovery (SfM), including: Using data from the GNSS receiver on board the UAV and the ground base station, the centimeter-level position X, Y, Z corresponding to each frame is calculated through PPK, providing 3 positional degrees of freedom; Simultaneously, grayscale image features are extracted and cross-frame matching is performed to construct a visual observation network, estimate the relative rotation and relative translation directions between adjacent frames, and provide pose change constraints. The centimeter-level positions X, Y, and Z are used as absolute constraints input into the SfM system. Combined with the posture change constraints of visual relative motion, a six-degree-of-freedom pose initial value is generated. The initial values ​​of the six degrees of freedom pose are jointly optimized by bundle adjustment to minimize the reprojection error. The final output is the six-DOF pose of each frame, including position and orientation.

[0016] Optionally, the step of combining camera calibration parameters, the six-degree-of-freedom pose, and the two-dimensional center coordinate sequence of the laser stripes, and using triangulation to back-project the two-dimensional coordinates of each frame along the imaging optical path, solving for their spatial intersection with the pre-calibrated laser plane, and generating the local three-dimensional point cloud corresponding to that frame, includes: Based on the calibrated camera intrinsic parameters and distortion coefficients, the two-dimensional center coordinate sequence in each frame of the image is subjected to distortion correction to eliminate the geometric deviation caused by lens imaging. Based on the six degrees of freedom pose, the corrected pixel is extended in reverse along its corresponding imaging optical path to construct a spatial ray originating from the camera optical center. By combining the spatial equation of the pre-calibrated line laser plane in the camera coordinate system, the unique intersection point of the spatial ray and the line laser plane is solved. Collect all intersection points to form the local 3D point cloud corresponding to the image frame.

[0017] A third aspect of this application provides a UAV line laser 3D reconstruction system, comprising: Unmanned aerial vehicles (UAVs) serve as carriers and provide aerial mobility. A line laser emitter, mounted on the UAV, is used to project a laser beam onto the target scene, forming laser stripes that change with the terrain. The camera integrates a sensor-memory-computing integrated chip as described in any one of the claims, for capturing the laser stripe image in real time and outputting a two-dimensional center coordinate sequence, and synchronously caching the corresponding grayscale image; An airborne synchronous acquisition node is used to receive the two-dimensional center coordinate sequence and synchronously record grayscale images, inertial measurement unit (IMU) data, and global navigation satellite system (GNSS) data. Ground-based post-processing equipment generates georegistered 3D point cloud models based on multi-source synchronous data from airborne synchronous acquisition nodes.

[0018] The integrated vision chip provided in this application achieves deep integration of perception and computation by mapping the processing array to the pixels of the image sensor and performing morphological skeleton extraction operations in parallel locally. This significantly shortens the data processing path, enabling the laser line center extraction process to be completed within ten microseconds. It effectively supports high-frequency acquisition and real-time response, breaking through the latency bottleneck of traditional perception, transmission, and computation architectures.

[0019] Other technical effects resulting from the additional features will be further illustrated in the corresponding embodiments. Attached Figure Description

[0020] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the structure of a sensor-memory-computing integrated vision chip according to an exemplary embodiment; Figure 2 This is a structural diagram of a heterogeneous line laser 3D reconstruction system for unmanned aerial vehicles based on a sensor-memory-computing integrated vision chip, according to an exemplary embodiment. Figure 3 This is a flowchart illustrating a heterogeneous line laser 3D reconstruction method for unmanned aerial vehicles based on a sensor-memory-computer integrated vision chip, according to an exemplary embodiment. Figure 4 A further flowchart illustrating a heterogeneous line laser 3D reconstruction method for unmanned aerial vehicles based on a sensor-memory-computer integrated vision chip according to an exemplary embodiment; Figure 5 This is a schematic diagram of 2d to 3d coordinate transformation calibration according to an exemplary embodiment, wherein (a) the diagram shows the camera intrinsic parameter calibration process; and (b) the diagram shows the process of fitting the spatial equation of the line laser plane in the camera coordinate system. Figure 6 This is a schematic diagram illustrating a laser line center extraction process based on parallel morphological computation according to an exemplary embodiment. Figure 7 The above describes a drone fixing structure according to an exemplary embodiment, wherein: (a), (b), and (c) represent three different structures; In the image: 1-Drone, 2-Camera, 3-Line laser emitter, 4-Raspberry Pi, 5-Navigation and positioning kit. Detailed Implementation

[0021] The present application will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any way. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present application, and these all fall within the protection scope of the present application. Parts not described in detail in the following embodiments can be implemented using existing technology.

[0022] Existing technologies suffer from real-time bottlenecks and high computational latency. Therefore, this application provides an integrated sensing, storage, and computing vision chip to address these problems.

[0023] like Figure 1As shown in some specific embodiments of this application, a sensing-memory-computing integrated chip includes a processing array. The processing array is composed of multiple processing units forming a two-dimensional array. Each processing unit corresponds one-to-one with a pixel in the input image and is used to perform morphological skeleton extraction operations through local parallel processing to identify the center path of a target in the image and output the two-dimensional center coordinate sequence of the target.

[0024] This application embodiment achieves deep integration of perception and computation by mapping the processing array to the pixels of the image sensor one-to-one and performing morphological skeleton extraction operations in parallel locally. This significantly shortens the data processing path, enabling the laser line center extraction process to be completed within ten microseconds, effectively supporting high-frequency acquisition and real-time response, and breaking through the latency bottleneck of the traditional "perception-transmission-computation" architecture.

[0025] like Figure 1 As shown, the processing unit in the above embodiments may include a pixel data cache module, a register group, a communication interface, an arithmetic logic unit, and a status flag bit, etc.

[0026] The Pixel Data Buffer (PIX) module is used to receive and buffer raw pixel data from the image sensor. The register file includes an analog register (AREG) and a digital register (DREG), which are used to store the analog values ​​of the grayscale image and the intermediate results after binarization, respectively. The communication interface (I / O & NEWS) supports data interaction with adjacent processing units on the North, East, South, and West sides, enabling data access in non-von Neumann neighborhoods. The Arithmetic Logic Unit (ALU) performs morphological skeleton extraction operations based on data from local registers and neighboring processing units. The status flag (FLAG) is used to indicate whether a data update has occurred within the current processing cycle and participates in the synchronization control during the global iteration process. The data processing flow of each component is as follows: the original pixel data is stored in the analog register (AREG) through the pixel data cache module (PIX), the arithmetic logic unit (ALU) combines local and neighboring data to complete the processing, and writes the result into the digital register (DREG), updates the status flag bit (FLAG), and after multiple iterations, outputs sparse two-dimensional center coordinates.

[0027] The embodiments described above in this application achieve pixel-level parallel computing within the processing unit through the collaboration of PIX, AREG / DREG and ALU, combined with the I / O & NEWS neighborhood access mechanism. This significantly improves the efficiency of morphological skeleton extraction, reduces data transfer overhead, and, in conjunction with FLAG, enables iterative synchronization control, effectively improving the system's real-time performance and energy efficiency.

[0028] To further address the power consumption bottleneck and the high demands on computing units, in some specific embodiments of this application, the chip also includes a main control unit. This main control unit issues unified instructions to the processing array, schedules the parallel computing tasks of the processing units, receives the processing results from each processing unit, and completes data aggregation, packaging, and output.

[0029] In the above embodiments, the main control unit uniformly schedules the parallel computing tasks of the processing array and only outputs sparse two-dimensional center coordinates, avoiding large-scale transmission of the original image, greatly reducing system bandwidth requirements and energy consumption, and the total power consumption can be as low as about 1.5 watts, significantly improving the energy efficiency ratio and continuous operation capability of the UAV payload.

[0030] Based on the same technical concept, another embodiment of this application provides a method for heterogeneous linear laser 3D reconstruction of unmanned aerial vehicles (UAVs) based on an integrated sensor-memory-computing vision chip, such as... Figure 3 and Figure 4 As shown, the three-dimensional reconstruction method includes the following steps: S100 projects a laser beam onto the target scene, forming laser stripes that change with the terrain; S200: Real-time capture of the laser stripe image formed by S100; parallel processing of the pixels of the laser stripe image to perform morphological skeleton extraction operation; identification of the center path of the target in the image; output of the two-dimensional center coordinate sequence of the target; and synchronous caching of the corresponding grayscale image. The S300 receives the two-dimensional center coordinate sequence from the S200 and simultaneously records grayscale images, inertial measurement unit (IMU) data, and Global Navigation Satellite System (GNSS) data.

[0031] S400 generates a georeferenced 3D point cloud model based on multi-source synchronous data obtained from S300.

[0032] The embodiments described above reduce transmission load and computational latency by using lightweight front-end processing based on centerline 2D coordinates and multi-source data synchronization, thereby achieving efficient and high-precision UAV 3D reconstruction and improving system real-time performance and energy efficiency.

[0033] Before the drone is used for the first time, a one-time static calibration is required to determine the fixed geometric relationships between the sensors. For example... Figure 5 As shown, in some embodiments of this application, the UAV is statically labeled using the following steps: First, camera intrinsic parameter calibration. Using a checkerboard calibration board, multiple checkerboard images with different poses were acquired, and the pixel coordinates of the corner points were extracted (corresponding to...). Figure 5 The pixel coordinate system in Figure (a) coordinates below Based on the physical dimensions of the checkerboard pattern, a perspective projection model from the world coordinate system to the pixel coordinate system is established: this model includes the pixel coordinate system. With image coordinate system (with main point) The scaling and offset transformations (with the origin as the reference point) and the perspective projection relationship from the image coordinate system to the camera coordinate system are then obtained. Through optimization, the intrinsic parameter matrix (including focal length) of the integrated sensing, storage, and computing camera is obtained. Principal point offset (e.g., parameters) and distortion coefficients.

[0034] Next, the extrinsic parameters of the camera and laser are calibrated. Images of the laser line projected onto the calibration plate are acquired from multiple angles, and the pixel coordinates of feature points on the laser line are extracted. Using the calibrated intrinsic parameters of the camera, the pixel coordinates (which will...) are then... Figure 5 The pixels in image (b) Convert to camera coordinate system Normalized coordinates are obtained; combined with the known pose of the calibration plate in the world coordinate system, the three-dimensional coordinates of the feature points on the laser line in the camera coordinate system are further obtained. (corresponding spatial point P( (The coordinates of ) are obtained. Plane fitting is performed on these 3D points, and the spatial equation of the line laser plane in the camera coordinate system is finally obtained as Ax+By+Cz+D=0.

[0035] Finally, the camera-IMU / GNSS extrinsic parameter calibration is performed, which calibrates the six-degree-of-freedom rigid body transformation relationship (rotation matrix and translation vector) between the camera coordinate system and the phase center of the IMU and GNSS antennas.

[0036] All calibration parameters are verified and stored in the airborne system for subsequent 3D reconstruction.

[0037] In the above embodiments of this application, the geometric constraint relationship between the camera, laser and navigation sensor is accurately established through system-level static calibration, providing high-precision initial parameters for subsequent 3D reconstruction and effectively improving the accuracy and stability of point cloud generation.

[0038] To ensure that the projected laser lines are clear and continuous, in some specific embodiments of this application, for S100, a straight laser beam is projected onto the target scene to form laser stripes that change with the terrain. This can be achieved by: The S101 line laser emitter is mounted on the bottom or lower front of the UAV, with its laser plane perpendicular to the flight direction and projected onto the ground at a certain angle.

[0039] S102, the drone flies at a constant speed along the predetermined route, and the laser continuously emits a fan-shaped light surface, forming a bright line on the ground that deforms with the undulation of the terrain.

[0040] The above embodiments can control the flight altitude and laser angle to ensure that the laser stripes clearly cover the target area and avoid occlusion or overexposure, thereby providing a high-quality active feature source for subsequent image capture and centerline extraction.

[0041] To achieve lightweight front-end computing and transmission, in some specific embodiments of this application, in S200, the laser stripe image formed in S100 is captured in real time, and morphological skeleton extraction is performed on the pixels of the laser stripe image in parallel to identify the center path of the target in the image and output the two-dimensional center coordinate sequence of the target. This can be achieved by the following steps: S201, perform binarization processing on the input image to obtain a binary image of the laser stripes; S202 performs a parallel erosion operation on the binary image to remove isolated bright pixels while preserving the main stripe structure. S203, enter the iterative refinement stage, use multiple sets of structural elements to perform hit-miss transformation on the main stripe structure in sequence, identify the current edge pixels, and set the matched edge points to zero in each iteration to achieve the layer-by-layer stripping of stripe boundaries. S204, repeat the above process until no pixel update occurs in a certain iteration, then determine that the skeleton has converged; S205, after the skeleton converges, outputs a sparse binary image containing only a single-pixel-width center line, and generates the corresponding two-dimensional center coordinate sequence.

[0042] The embodiments described above in this application, by implementing parallel binarization, erosion, and iterative skeleton extraction on an integrated sensing, memory, and computing chip, output only sparse center coordinates, significantly reducing data volume and computational latency. This avoids the transmission of the original image, significantly reducing bandwidth pressure and power consumption, improving system real-time performance, meeting the high-frequency acquisition requirements of UAVs during high-speed flight, and providing low-redundancy, high-quality feature input for high-precision reconstruction in the backend.

[0043] Furthermore, in some specific embodiments, S203 enters the iterative refinement stage, where multiple sets of structural elements are sequentially subjected to hit-miss transformation on the main stripe structure to identify and peel off edge pixels. In each iteration, the matched edge points are synchronously set to zero, achieving layer-by-layer thinning of the stripe boundaries. Figure 6 As shown, it includes the following steps: After image binarization and parallel erosion denoising, the system enters an iterative refinement stage, gradually compressing the laser stripes of a certain width into a single-pixel-wide central skeleton line. The specific process is as follows: S2031, Initialize the processing object.

[0044] The input is a denoised binary image, where white areas represent laser stripes (value 1) and black areas represent the background (value 0). The stripes still have a width of multiple pixels, and the edges need to be peeled off layer by layer iteratively.

[0045] S2032, Prepare the structural element group.

[0046] A set of 8 3×3 structuring elements (SEs) is pre-defined, divided into two groups: The first group (a1~a4): used to detect the foreground boundary in a certain direction; The second group (b1~b4): is used to ensure that the corresponding reverse area has a blank background.

[0047] Each pair (ai, bi) constitutes a "hit-and-miss transform" (HMT) template, which matches edge features in different directions (such as east, south, west, north, and diagonal directions).

[0048] Specifically, "hitting" refers to whether the expected bright spot exists in the foreground area (i.e., the area that should be bright) within a specific neighborhood of the current pixel; "missing" refers to whether the bright pixel does not actually appear in the background area (i.e., the part that should not have light) in the corresponding reverse area.

[0049] The Hit-or-Miss Transform (HMT) is a binary image pattern recognition method based on mathematical morphology. It uses a pre-defined pair of structuring element templates to define regions that "must be foreground" and regions that "must be background," enabling precise localization of specific geometric structures (such as edges, endpoints, or corners) in an image. Its core idea is that a pixel is considered a target feature point only when both conditions are met: "it should be illuminated" (hit) and "it shouldn't be illuminated" (miss). This dual-constraint mechanism effectively improves the accuracy and noise resistance of edge recognition, ensuring shape preservation, centering, and stable convergence during skeleton extraction.

[0050] S2033 executes a complete iteration.

[0051] In each iteration, these eight structuring elements are applied sequentially to scan the entire image: For each structuring element combination (ai, bi), all processing units (PEs) perform an HMT operation in parallel once. If a pixel satisfies the condition that "it is currently in the foreground and its neighborhood conforms to the edge pattern", it is marked as an edge point that can be deleted. All marked edge points are synchronously set to zero by the corresponding processing unit, realizing parallel edge trimming operation at the whole image level.

[0052] S2034, Update the status and determine whether to continue.

[0053] Each processing unit updates its local status flag based on whether its output has changed; The main control unit detects whether a status flag bit is set. If no pixels are modified after traversing all structural elements in this round, the algorithm is considered to have converged; otherwise, return to S2033 and start the next round of iteration.

[0054] S2035, Termination Condition and Output.

[0055] The loop terminates when no pixel changes occur after a round of traversing all structural elements.

[0056] The remaining white pixels form a continuous, centered, single-pixel-width skeleton line, representing the geometric center path of the laser stripe.

[0057] Finally, the coordinates of the skeleton point are extracted to form a sparse two-dimensional center coordinate sequence for subsequent transmission and three-dimensional reconstruction.

[0058] The above embodiments apply directionally sensitive structural elements through multiple rounds of iteration, combined with neighborhood communication between processing units and parallel execution of hit-miss transformation, to synchronously complete edge recognition and pixel zeroing operations within each clock cycle, thereby achieving layer-by-layer stripping of laser stripe boundaries.

[0059] The entire skeleton extraction process is completed in situ on the image sensor, avoiding the extensive transmission and centralized computation of the original image, significantly reducing system latency and power consumption. Global convergence detection is performed using status flags (0 or 1) to ensure the algorithm terminates stably in the shortest possible time, ultimately outputting a precise single-pixel width centerline coordinate sequence. Thanks to its highly parallel in-memory computing architecture, this method can complete all processing within a single frame exposure time, meeting the real-time feature extraction requirements of high-speed UAV flight scenarios, and further overcoming the real-time bottlenecks and high computational latency technical barriers inherent in traditional 3D reconstruction.

[0060] In some specific embodiments of this application, in step S300, receiving the real-time 2D coordinate data stream and simultaneously recording the grayscale image, IMU data, and airborne GNSS raw data for the motion recovery structure SfM can be achieved through the following steps: S301, the Raspberry Pi module receives a sparse two-dimensional center coordinate sequence from the output of the integrated vision chip (CMP) via a high-speed serial interface, as a lightweight feature data stream for laser stripes; S302 simultaneously triggers or subscribes to grayscale image frames at corresponding times, which are then transmitted from the camera or auxiliary imaging unit to the Raspberry Pi and stored for feature matching and pose optimization in subsequent SfM. The S303 achieves time synchronization through hardware interrupts or PPS (pulse per second) signals, ensuring that the 2D coordinates, grayscale images, IMU data, and GNSS observations output by the camera have a unified time reference. S304 continuously acquires raw acceleration and angular velocity data from the inertial measurement unit (IMU) at a high frequency (≥100Hz), records its timestamps, and caches them; S305 synchronously acquires the raw observation data (including pseudorange, carrier phase, etc.) output by the airborne GNSS module and stores it together with the antenna position information; S306: All data is encapsulated and cached locally using a unified timestamp to form structured data packets, which are then uploaded to ground post-processing equipment for post-processing after the flight.

[0061] The embodiments described above in this application ensure spatiotemporal alignment of 2D coordinates, images, and navigation data through synchronous acquisition of multi-source data and unified timestamp encapsulation, providing a reliable data foundation for high-precision pose calculation and 3D reconstruction.

[0062] In some specific embodiments of this application, in step S400, three-dimensional reconstruction is performed based on the synchronized two-dimensional center coordinate sequence, grayscale image, and raw IMU and GNSS data. This can be achieved through the following two steps: S401, a ground station or offline computing device, receives multi-source synchronous data output by Raspberry Pi and obtains high-precision six-DOF pose for each frame of image through joint calculation using post-differential positioning (PPK) and structure of motion restoration (SfM) bundle adjustment. S402, based on the two-dimensional center coordinates extracted by the camera, the camera calibration parameters, and the calculated six-degree-of-freedom pose, uses triangulation to back-project the 2D coordinates of each frame into a local three-dimensional point cloud. S403, after bilateral filtering and denoising, is registered to a unified global coordinate system based on the timestamp and pose information of the corresponding frame, generating a complete and geo-registered 3D point cloud model.

[0063] The embodiments described above in this application obtain high-precision pose through joint calculation of PPK and SfM, and achieve point cloud registration without ICP by combining triangulation and rigid body transformation, which significantly improves the accuracy, efficiency and geographic consistency of 3D reconstruction.

[0064] Furthermore, in some specific embodiments, S401, configured at a ground station or offline computing device, receives multi-source synchronization data output by a Raspberry Pi, and obtains high-precision six-DOF pose for each frame of image through joint calculation using post-differential positioning (PPK) and structure of motion restoration (SfM) bundle adjustment; the following steps can be adopted: S4011 first uses the observation data from the GNSS receiver on the UAV and the ground base station, and then uses post-differential positioning (PPK) technology to calculate the centimeter-level position sequence of the UAV's flight trajectory.

[0065] Specifically, PPK (Post-Processed Kinematic) is a high-precision satellite navigation and positioning technology, belonging to the category of GNSS differential positioning.

[0066] This step involves comparing the GNSS data on the UAV with data from a known ground-based base station after the fact to eliminate common errors such as atmospheric delay and satellite clock bias, thereby obtaining centimeter-level precise positioning at every moment along the UAV's flight path.

[0067] S4012 simultaneously extracts feature points from the grayscale image and performs cross-frame matching to construct a visual observation network for estimating the relative motion of the camera.

[0068] Specifically, by identifying and tracking salient feature points in continuous images, establishing cross-frame correspondences, forming a visual observation network, analyzing the motion trends of these points, and calculating the relative displacement and attitude changes of the camera between adjacent moments, motion constraint information can be provided for subsequent 3D reconstruction.

[0069] S4013 uses the high-precision position provided by S4011PPK as the initial constraint and inputs it into the SfM system to assist in recovering the rotational attitude angle of each frame of image; Specifically, SfM (Structure from Motion) is a 3D reconstruction technique based on multi-view geometry, and it is a mature method in the fields of computer vision and photogrammetry.

[0070] S4014 uses bundle adjustment to globally optimize all parameters (position information X, Y, Z obtained from S4011 and rotation attitude angle obtained from S4013), and jointly adjusts the camera pose and 3D point coordinates to minimize geometric projection error. Specifically, bundle adjustment (BA) is a key global optimization technique in photogrammetry and computer vision. Its core objective is: Simultaneously optimize camera pose and 3D spatial point coordinates to minimize the reprojection error between feature points and their projection positions in all observed images.

[0071] The S4015 ultimately outputs the high-precision six-DOF pose corresponding to each frame of the image, including the precise position (X, Y, Z) and attitude rotation angles (roll, pitch, yaw).

[0072] The above embodiments provide high-precision initial position values ​​through PPK, and combine SfM and bundle adjustment for joint optimization, effectively improving the accuracy and stability of camera pose calculation, and providing a reliable spatiotemporal reference for subsequent 3D point cloud reconstruction.

[0073] Furthermore, in some specific embodiments, S402, based on the two-dimensional center coordinates extracted by the camera, the camera calibration parameters, and the calculated six-degree-of-freedom pose, the 2D coordinates of each frame are back-projected into a local three-dimensional point cloud using triangulation, and the following steps are taken: S4021 performs distortion correction on the two-dimensional coordinates of the center line in each frame of the image based on the calibrated camera intrinsic parameters and distortion coefficients, thereby eliminating geometric deviations caused by lens imaging. S4022, based on the six-degree-of-freedom pose determined by S401, extends the corrected pixel along its corresponding imaging optical path in the reverse direction to construct a spatial ray originating from the camera optical center; Specifically, the direction of the imaging optical path is determined by both camera intrinsic parameters and pixel coordinates: based on camera calibration parameters, two-dimensional pixels in the image can be mapped to local direction vectors originating from the camera's optical center; the actual starting point and spatial orientation of this ray in global space are determined by the corresponding six-degree-of-freedom pose (i.e., position and orientation, or EO parameters). Only by combining the aforementioned local geometric relationships and global pose information can two-dimensional pixels be accurately back-projected into an effective observation ray in three-dimensional space, which can then be used for subsequent spatial intersection calculations with the pre-calibrated line laser plane.

[0074] S4023, combining the spatial equation of the pre-calibrated line laser plane in the camera coordinate system, solve for the unique intersection point between the spatial ray and the line laser plane obtained in S4022; S4024: Collect all intersection points to form the local 3D point cloud corresponding to the image frame.

[0075] The embodiments described above in this application use triangulation combined with calibration parameters and laser plane constraints to accurately back-project the 2D center coordinates into a local three-dimensional point cloud, achieving high-precision, low-error single-frame three-dimensional information recovery.

[0076] Furthermore, in some specific implementations, S403, after bilateral filtering and denoising, the model is registered to a unified global coordinate system based on the timestamp and pose information of the corresponding frame to generate a complete and geo-registered 3D point cloud model, including the following steps: Immediately use its corresponding EOE pose Through a rigid body coordinate transformation ( 9. Directly integrate it into the global coordinate system. This method eliminates the need for ICP registration, thus ensuring high-precision and high-efficiency point cloud fusion.

[0077] In the above embodiments, this method does not require ICP registration, thus ensuring high-precision and high-efficiency point cloud fusion.

[0078] Based on the same technical concept, another embodiment of this application also provides a heterogeneous line laser 3D reconstruction system for unmanned aerial vehicles based on a sensor-memory-computing integrated vision chip, such as... Figure 2 As shown, it includes a drone 1, a line laser emitter 3, an IVP camera 4, an airborne synchronous acquisition node, and ground post-processing equipment.

[0079] The drone 1 serves as the carrier and provides aerial mobility; A line laser emitter 3 is mounted on the UAV 1 to project a laser beam onto the target scene, forming laser stripes that change with the terrain. IVP camera 2, integrated with such Figure 1The integrated sensing, storage, and computing chip in the illustrated embodiment is used to capture laser stripe images in real time and output a two-dimensional center coordinate sequence, while simultaneously caching the corresponding grayscale images. Airborne synchronous acquisition nodes are used to receive two-dimensional center coordinate sequences and simultaneously record grayscale images, inertial measurement unit (IMU) data, and Global Navigation Satellite System (GNSS) data.

[0080] Specifically, the airborne synchronous acquisition node can be a Raspberry Pi, a Jetson series module, or an industrial-grade embedded computer. It connects to the sensor of the IVP camera itself, receives its laser stripe image, and processes it into a grayscale image.

[0081] Ground-based post-processing equipment generates georegistered 3D point cloud models based on multi-source synchronous data from airborne synchronous acquisition nodes.

[0082] Specifically, the ground post-processing equipment can be a laptop, desktop computer, or server; there are no restrictions on the specific form, and it can be flexibly configured according to actual needs.

[0083] This application addresses the technical challenges of poor real-time performance, high data transmission load, high power consumption, and difficulty in modeling weakly textured scenes in traditional UAV 3D reconstruction by integrating a sensor-in-memory (SIMM) integrated vision chip with a heterogeneous computing architecture. The system utilizes a CMP chip to perform parallel skeleton extraction of laser stripes at the front end, outputting only sparse two-dimensional coordinates, significantly reducing bandwidth requirements and processing latency. Combined with line laser active illumination, geometric features are enhanced, improving the reconstruction accuracy of textureless regions. Simultaneous acquisition of multi-source data via an onboard synchronous acquisition node ensures the spatiotemporal consistency of pose calculation and point cloud fusion at the back end, significantly reducing system power consumption while maintaining high accuracy. This makes it suitable for UAV operations requiring long endurance and high-frequency sampling.

[0084] The following examples and comparative examples will be used to further illustrate this application in order to better understand the above-mentioned technical solutions. It should be understood that the following are only some examples and are not intended to limit this application.

[0085] Application Example 1: A heterogeneous line laser 3D reconstruction system for unmanned aerial vehicles (UAVs) based on a sensor-memory-computing integrated vision chip is constructed, specifically including the following components: The integrated camera 2 includes an image sensor array, a processing unit array (PE), a pixel-level light sensor circuit (PIX), an analog register (AREG), a digital register (DREG), an arithmetic logic unit (ALU), a neighborhood communication interface (NEWS), a status flag (FLAG), a bias circuit, a clock drive, an analog-to-digital converter, and a main control MCU.

[0086] Drone 1: It uses a DJI Matrice M350 RTK and is equipped with an E-Port expansion interface and a six-way sensing system.

[0087] Navigation and Positioning Suite 5: Includes an airborne GNSS receiver, a ground GNSS reference station, and an inertial measurement unit (IMU).

[0088] Line laser emitter 3: A green laser module with a wavelength of 532nm and an output power of 400mW, supporting DC power supply of 2.8–3.7V and featuring a magnetic mounting structure.

[0089] Onboard computer: Composed of Raspberry Pi 4 (Raspberry Pi 5), 5V / 5A voltage regulator module and 4G / 5G communication module.

[0090] 3D printed mounting plate (e.g.) Figure 7 As shown, the specific structure can be adjusted according to the actual situation, such as (a), (b) and (c) three options: made of high-strength resin material, 4mm thick, with a one-piece molded structure, and has custom slots and fixing holes for mounting the sensor camera, line laser emitter and airborne computer.

[0091] Ground station post-processing equipment: A laptop equipped with an Intel i7 processor and 16GB of memory, running self-developed 3D reconstruction software.

[0092] The implementation process is briefly described as follows: Perform static calibration before the system is used for the first time: Use a checkerboard calibration board to complete the calibration of the in-camera distortion coefficients. By projecting lasers at multiple angles onto a calibration plate, the equation of the line laser plane is fitted (Ax+By+Cz+D=0). The six-degree-of-freedom extrinsic parameter relationship between the phase center of the camera and the IMU / GNSS antenna is calibrated, and the parameters are stored in the airborne system.

[0093] Perform flight missions: The UAV 1 flies at a constant speed along a predetermined route, and the line laser projects a straight light stripe onto the ground; Camera 2 captures laser images in real time, performs morphological skeleton extraction on-chip in parallel, and outputs sparse two-dimensional center coordinates for each frame; The Raspberry Pi 4 synchronously receives 2D coordinate streams and records the corresponding grayscale image, IMU and GNSS raw data. All data is timestamped according to the PPS signal.

[0094] Post-processing stage: After the flight is completed, the data will be uploaded to the ground station; PPK is used to calculate the centimeter-level drone trajectory, and SfM bundle adjustment is used to optimize the six-DOF pose of each frame image; Based on pose, camera parameters and laser plane constraints, the 2D center point of each frame is back-projected into a local 3D point cloud through triangulation. Based on the EOE pose Tworld→camerai, a rigid body transformation is performed (Pglobal,i=Tworld→camerai×Plocal,i), and the model is directly registered to the global coordinate system to generate a fully georegistered 3D point cloud model.

[0095] The preferred features in the above embodiments can be used individually in any embodiment, or in any combination thereof, provided they do not conflict with each other. Furthermore, parts not described in detail in the embodiments can be implemented using existing technologies.

[0096] In the description of the embodiments of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.

[0097] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature.

[0098] In the description of the embodiments in this application, "multiple" means two or more, unless otherwise explicitly specified. In this application, unless otherwise explicitly specified and limited, the terms "installed," "connected," "linked," "fixed," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0099] The terms "comprising" and "having," and any variations thereof, in the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or devices.

[0100] The foregoing has described some specific embodiments of this application. It should be understood that this application is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the substantive content of this application. The above-described preferred features can be used in any combination without conflict.

Claims

1. A sensing-memory-computing integrated chip, characterized in that, It includes a processing array, which is a two-dimensional array composed of multiple processing units. Each processing unit corresponds to a pixel of the input image, senses the original data of the pixel, and stores it locally. Each processing unit accesses the data stored in itself and adjacent processing units to perform morphological skeleton extraction operations in parallel, identify the center path of the target in the image, and output the two-dimensional center coordinate sequence of the target.

2. The integrated sensing, memory, and computing chip according to claim 1, characterized in that, It also includes a main control unit, which is used to issue unified instructions to the processing array, schedule the parallel computing tasks of each processing unit, receive the processing results, and complete the aggregation, packaging and output of data.

3. The integrated sensing, storage, and computing chip according to claim 1, characterized in that, The processing unit includes: A pixel data caching module is used to receive and cache raw pixel data from the image sensor; An analog register is used to store the raw pixel data; The communication interface supports data interaction with adjacent processing units in the north, east, south, and west, enabling neighborhood access in non-von Neumann architectures. The arithmetic logic unit performs morphological skeleton extraction operations based on local analog register data and neighborhood data; Digital registers are used to store intermediate calculation results of arithmetic logic units; The status flag is used to indicate whether the processing unit has undergone a value update within the current processing cycle.

4. A method for line laser 3D reconstruction of unmanned aerial vehicles (UAVs), characterized in that, The integrated sensing, memory, and computing chip according to claim 1 or 2 includes: A laser beam is projected onto the target scene, forming laser stripes that change with the terrain; The laser stripe image is captured in real time, and morphological skeleton extraction is performed on the pixels of the laser stripe image in parallel to identify the center path of the target in the image, output the two-dimensional center coordinate sequence of the target, and simultaneously cache the corresponding grayscale image. Receive the two-dimensional center coordinate sequence and simultaneously record grayscale images, inertial measurement unit (IMU) data, and global navigation satellite system (GNSS) data; A georeferenced 3D point cloud model is generated based on multi-source synchronous data.

5. The UAV line laser three-dimensional reconstruction method according to claim 4, characterized in that, The parallel pixel processing of the laser stripe image performs morphological skeleton extraction to identify the center path of the target in the image and outputs the two-dimensional center coordinate sequence of the target, including: The input laser stripe image is binarized to obtain a binary image of the laser stripes; Perform a parallel erosion operation on the binary image to remove isolated bright pixels while retaining the main stripe structure; In the iterative refinement stage, multiple sets of structural elements are used to sequentially perform hit-miss transformation on the main stripe structure to identify the current edge pixels. In each iteration, the identified edge points are synchronously set to zero to achieve the layer-by-layer peeling of the boundary of the main stripe structure. Repeat the above process until no processing unit state update occurs in a certain iteration, at which point the skeleton is considered to have converged. After the skeleton converges, the output is a sparse binary image containing only a single-pixel-width center line, and the corresponding two-dimensional center coordinate sequence is generated.

6. The UAV line laser three-dimensional reconstruction method according to claim 5, characterized in that, The iterative refinement stage involves sequentially performing hit-and-miss transformations on multiple sets of structural elements to identify current edge pixels. In each iteration, the identified edge points are synchronously set to zero, achieving layer-by-layer peeling of the boundaries of the main stripe structure. This includes: Enter the loop iteration process, and in each round, call multiple pre-set fixed structure elements in sequence; For each set of structure elements, all processing units perform the following operations in parallel: Centered on the corresponding pixel, the binarized states of the adjacent processing units in the north, east, south, and west are obtained through the communication interface to form 4-neighborhood data; By combining local pixel values ​​with neighborhood data, it is determined whether the pixel matches the edge pattern defined by the current structuring element. If it matches, the pixel is determined to be a removable edge point. All processing units that are identified as edge points synchronously set their output to zero, thereby achieving parallel edge trimming of the image boundary; After each round, each processing unit updates its local status flag based on whether its value has changed; When all structural elements have been fully traversed in a certain round, and the status flags of all processing units have not been set, the iteration terminates and the skeleton extraction is determined to be complete.

7. The UAV line laser three-dimensional reconstruction method according to claim 4, characterized in that, The three-dimensional reconstruction based on the synchronized two-dimensional center coordinates data stream, image information, and orientation information includes: Based on inertial measurement unit (IMU) and global navigation satellite system (GNSS) data, combined with sparse feature matching of grayscale images, the six-degree-of-freedom pose of each frame image is calculated through joint optimization of post-differential positioning (PPK) and structure for motion recovery (SfM). Combining the camera calibration parameters, the six-degree-of-freedom pose, and the two-dimensional center coordinate sequence of the laser stripes, the two-dimensional coordinates of each frame are back-projected along the imaging optical path using the triangulation method, and the spatial intersection points with the pre-calibrated laser plane are solved to generate the local three-dimensional point cloud corresponding to that frame. Based on the timestamps and six-degree-of-freedom poses of each frame, all local 3D point clouds are registered to a unified global coordinate system through rigid body transformation, and then fused to generate a complete and geo-registered 3D point cloud model.

8. The UAV line laser three-dimensional reconstruction method according to claim 7, characterized in that, The method, based on inertial measurement unit (IMU) and global navigation satellite system (GNSS) data, combined with sparse feature matching of grayscale images, and through joint optimization using post-differential positioning (PPK) and structure-of-motion (SfM) modeling, calculates the six-degree-of-freedom pose of each frame, including: Using data from the GNSS receiver on board the UAV and the ground base station, the centimeter-level position X, Y, Z corresponding to each frame is calculated through PPK, providing 3 positional degrees of freedom; Simultaneously, grayscale image features are extracted and cross-frame matching is performed to construct a visual observation network, estimate the relative rotation and relative translation directions between adjacent frames, and provide pose change constraints. The centimeter-level positions X, Y, and Z are used as absolute constraints input into the SfM system. Combined with the posture change constraints of visual relative motion, a six-degree-of-freedom pose initial value is generated. The initial values ​​of the six degrees of freedom pose are jointly optimized by bundle adjustment to minimize the reprojection error. The final output is the six-DOF pose of each frame, including position and orientation.

9. The UAV line laser three-dimensional reconstruction method according to claim 7, characterized in that, The method combines camera calibration parameters, the six-degree-of-freedom pose, and the two-dimensional center coordinate sequence of the laser stripes. Triangulation is used to back-project the two-dimensional coordinates of each frame along the imaging optical path, solve for their spatial intersection with the pre-calibrated laser plane, and generate the corresponding local three-dimensional point cloud for that frame. This includes: Based on the calibrated camera intrinsic parameters and distortion coefficients, the two-dimensional center coordinate sequence in each frame of the image is subjected to distortion correction to eliminate the geometric deviation caused by lens imaging. Based on the six degrees of freedom pose, the corrected pixel is extended in reverse along its corresponding imaging optical path to construct a spatial ray originating from the camera optical center. By combining the spatial equation of the pre-calibrated line laser plane in the camera coordinate system, the unique intersection point of the spatial ray and the line laser plane is solved. Collect all intersection points to form the local 3D point cloud corresponding to the image frame.

10. A UAV line laser 3D reconstruction system, characterized in that, include: Unmanned aerial vehicles (UAVs) serve as carriers and provide aerial mobility. A line laser emitter, mounted on the UAV, is used to project a laser beam onto the target scene, forming laser stripes that change with the terrain. The camera integrates a sensor-memory-computing integrated chip as described in any one of claims 1–3, for capturing the laser stripe image in real time and outputting a two-dimensional center coordinate sequence, and synchronously caching the corresponding grayscale image; An airborne synchronous acquisition node is used to receive the two-dimensional center coordinate sequence and synchronously record grayscale images, inertial measurement unit (IMU) data, and global navigation satellite system (GNSS) data. Ground-based post-processing equipment generates georegistered 3D point cloud models based on multi-source synchronous data from airborne synchronous acquisition nodes.

Images