A mobile terminal geometry model construction method and device

Abstract

The application provides a mobile terminal geometric structure model construction method and device, and relates to the technical field of computer vision. The method comprises the following steps: acquiring image data collected by a mobile terminal and associated motion sensing data; correcting the image data based on the motion sensing data; extracting features of the corrected image data through a lightweight convolutional neural network, and adaptively adjusting the calculation overhead of feature extraction based on the device running state to obtain a geometric structure feature set; and performing structure completion and fusion processing on the geometric structure feature set based on geometric constraint rules to generate a geometric structure model. The application can realize real-time geometric structure model construction with high precision and high completeness on a mobile terminal with limited resources by correcting through the fusion of motion sensing data, adapting the mobile terminal computing power by combining a dynamic attention mechanism, and completing the missing structure by using geometric rules, thereby effectively improving the robustness of mobile terminal augmented reality and three-dimensional measurement.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method and apparatus for constructing a geometric structure model for a mobile device. Background Technology

[0002] In applications such as mobile augmented reality, structural measurement, and 3D reconstruction, accurate and real-time perception and construction of geometric models of the physical environment are critical requirements. However, directly constructing geometric models on mobile devices faces numerous challenges. First, handheld devices are prone to image jitter, resulting in blurred and geometrically distorted images, severely impacting the accuracy of subsequent geometric feature extraction and model construction. Second, mobile devices have limited computing resources (such as GPU power), making it difficult to directly deploy computationally intensive traditional structural perception models. Traditional solutions typically rely on high-performance servers for data processing, which cannot meet the requirements for real-time geometric model construction on mobile devices, or severely sacrifice perception accuracy when computing power is limited. Therefore, there is an urgent need for a technical solution that can effectively overcome image jitter interference, adapt to limited computing power, and achieve real-time construction of complete geometric models under the resource constraints of mobile devices. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] Therefore, the first aspect of this application proposes a method for constructing a mobile terminal geometric structure model, including: Acquire image data collected by the mobile device and motion sensing data associated with the image data; The image data is corrected based on the motion sensing data to obtain corrected image data. The corrected image data is feature extracted using a lightweight convolutional neural network, and the computational cost of feature extraction is adaptively adjusted based on the device's operating status during the extraction process to obtain the geometric structure feature set output by the lightweight convolutional neural network. Based on preset geometric constraint rules, the geometric structure feature set is subjected to structural completion and fusion processing to generate a geometric structure model.

[0005] In some embodiments of this application, the step of correcting the image data based on the motion sensing data to obtain corrected image data includes: generating a motion blur kernel based on the motion sensing data; and using the motion blur kernel to deblur the image data to obtain corrected image data.

[0006] In some embodiments of this application, the computational overhead of feature extraction is adaptively adjusted based on the device operating status during the extraction process to obtain the geometric structure feature set output by the lightweight convolutional neural network. This includes: acquiring device operating status information, which includes processor utilization and / or battery power; determining the currently available computational overhead level based on the device operating status information; selecting a feature extraction algorithm that matches the computational overhead level; performing feature extraction on the corrected image data to obtain the geometric structure feature set.

[0007] In some embodiments of this application, determining the currently available computing overhead level based on the device operating status information includes: determining the processor idle capacity based on the processor utilization rate in the device operating status information; determining the remaining battery life ratio based on the battery power in the device operating status information; calculating a weighted comprehensive resource index based on the processor idle capacity and the remaining battery life ratio, and mapping the weighted comprehensive resource index to a preset computing overhead level.

[0008] In some embodiments of this application, the step of performing structural completion and fusion processing on the geometric structure feature set based on preset geometric constraint rules to generate a geometric structure model includes: determining the missing geometric structure regions in the geometric structure feature set; performing structural calculation on the missing geometric structure regions according to the geometric constraint rules to generate a completed geometric structure; and fusing the completed geometric structure with the initial geometric structure in the geometric structure feature set to generate the geometric structure model.

[0009] In some embodiments of this application, the step of performing structural deduction on the missing geometric structure region according to the geometric constraint rules to generate a complete geometric structure includes: extracting boundary geometric features adjacent to the missing geometric structure region from the geometric structure feature set; extending and constraining the boundary geometric features based on the geometric constraint rules to deduce the geometric topology of the missing region; and converting the deduced geometric topology into the complete geometric structure.

[0010] In some embodiments of this application, the step of acquiring image data collected by a mobile terminal and motion sensing data associated with the image data includes: acquiring image data through an image acquisition device on the mobile terminal and synchronously acquiring the motion sensing data through an inertial measurement unit; and aligning the image data and the motion sensing data with timestamps to obtain the image data and motion sensing data associated with the image data.

[0011] A second aspect of this application provides a mobile terminal geometric structure model construction device, comprising: The acquisition module is used to acquire image data collected by the mobile terminal and motion sensing data associated with the image data; The processing module is used to perform correction processing on the image data based on the motion sensing data to obtain corrected image data; The extraction module is used to extract features from the corrected image data using a lightweight convolutional neural network, and adaptively adjust the computational cost of feature extraction based on the device operating status during the extraction process to obtain the geometric structure feature set output by the lightweight convolutional neural network. The generation module is used to perform structural completion and fusion processing on the geometric feature set based on preset geometric constraint rules to generate a geometric structure model.

[0012] A third aspect of this application provides an electronic device, including: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method described in the first aspect above.

[0013] A fourth aspect of this application provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method described in the first aspect above.

[0014] The mobile geometric structure model construction method provided in this application simultaneously acquires images and motion sensing data on the mobile device and correlates them over time. Based on the motion sensing data, the image is corrected, eliminating jitter interference. Adaptive noise reduction and resolution standardization significantly reduce the data volume while preserving geometric edges. By using a lightweight network and adaptively switching computing modes according to the device's real-time operating status, it can balance the accuracy and real-time performance of feature extraction under resource constraints. Multi-scale fusion and structural completion based on preset geometric constraint rules can repair structural breaks caused by occlusion, texture loss, etc., generating a complete and highly accurate target geometric structure model. This application can, to a certain extent, solve the problems of mobile geometric structure perception being susceptible to jitter interference, requiring high computing power, and incomplete perception in complex scenarios such as occlusion. It achieves highly robust and high-precision real-time geometric structure perception and modeling on resource-constrained mobile terminals.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a method for constructing a mobile terminal geometric structure model, provided in an embodiment of this application; Figure 2 This is a schematic diagram of a mobile terminal geometric structure model construction device provided in an embodiment of this application. Detailed Implementation

[0017] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0018] Specifically, the mobile terminal geometric structure model construction method and apparatus of this application embodiment are described below with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart illustrating a method for constructing a mobile terminal geometric structure model, as provided in an embodiment of this application. Figure 1 As shown, the method for constructing the mobile device's geometric structure model may include the following steps: Step 101: Acquire image data collected by the mobile device and motion sensing data associated with the image data.

[0020] The mobile device can acquire raw image data through built-in or external image acquisition devices, such as continuously acquiring scene image frames at a default resolution of 1920×1080 pixels in RGB format. Simultaneously, motion sensing data is acquired through an inertial measurement unit integrated within the mobile terminal. This motion sensing data includes, but is not limited to, real-time angular velocity data around the X and Y axes output by the gyroscope and linear motion acceleration data output by the accelerometer. The sampling frequency is set to 100Hz to ensure the accuracy of the correspondence between motion data and image frame exposure times. The acquired raw image data and motion sensing data are constructed in real-time into a data input queue (DataQueue). Each frame of image data is associated with and stored with the motion sensing data read at the corresponding timestamp, forming an "image-motion data pair" for direct use in subsequent correction processing steps.

[0021] In some embodiments of this application, image data can be acquired via an image acquisition device on a mobile device, and motion sensing data can be acquired synchronously via an inertial measurement unit. The image data and motion sensing data are then time-stamped to obtain the image data and the motion sensing data associated with it. The image acquisition device can be a monocular camera, a binocular camera, or an RGB camera. The camera is configured to acquire a sequence of color or depth images of the environment. The inertial measurement unit integrates an accelerometer and a gyroscope, and is capable of outputting three-dimensional acceleration and three-dimensional angular velocity information. To ensure temporal correspondence between images and motion sensing data, the image acquisition device and the inertial measurement unit can perform synchronized acquisition through a unified hardware trigger signal or a shared system clock. For example, at the start of exposure of each frame of image, the IMU is simultaneously triggered to generate a sampling pulse, or a microcontroller provides a synchronized external clock reference for the two types of sensors, ensuring that each frame of image has a record of IMU measurement data that strictly corresponds in time.

[0022] Since the image acquisition frame rate and the IMU sampling frequency are usually different, and delays may be introduced during transmission, time sequence alignment processing is performed on the originally acquired data. Based on the synchronized timestamps respectively carried by the image data and the IMU data, a linear interpolation or nearest neighbor matching algorithm is adopted to match each frame of image to the closest IMU measurement value or the interpolated IMU result. For example, if the timestamp of an image frame is t_img, and the IMU data stream contains {t1, IMU1} and {t2, IMU2} where t1 < t_img ≤ t2, the interpolated motion data IMU_img is calculated via the time weight coefficient α = (t_img - t1) / (t2 - t1) as IMU_img = (1-α)·IMU1 + α·IMU2. After alignment, each frame of image is provided with a set of synchronized motion sensing data, forming associated data pairs for subsequent processing. Accordingly, "acquiring image data and motion sensing data associated with the image data" in step 101 of the independent claim is defined as a high-precision, reproducible specific implementation.

[0023] To adapt to the differences between different models of mobile terminal hardware, in some embodiments, the working states and supported parameters of the camera sensor and the inertial sensor can be automatically detected in the initialization stage. If a multi-camera configuration is detected, multi-channel image acquisition is synchronously started, and each channel of image is individually associated with the same set of motion sensing data, so as to provide spatiotemporally consistent basic data for subsequent multi-view fusion. By collaboratively acquiring and associating image and motion sensor data locally on the mobile terminal, this step not only provides high-precision motion reference information for jitter compensation based on the inertial sensor, but also ensures the spatiotemporal consistency of subsequent correction operations through the time synchronization mechanism, which is a prerequisite for ensuring the low-latency and high-robustness operation of the entire method on the mobile terminal.

[0024] Step 102: performing correction processing on the image data based on the motion sensing data to obtain corrected image data.

[0025] In some embodiments of this application, angular velocity information from motion sensing data can be used for inverse translational jitter compensation. Based on the angular velocity data around the X and Y axes measured by the gyroscope, the jitter offset generated by the device during image acquisition is calculated by trajectory integration over the exposure time. This is achieved using the motion trajectory integration formula. , Where T is the exposure time, T=0.01s, the x and y axis offset values ​​are obtained. , These represent the real-time angular velocity data around the x-axis and y-axis collected by the gyroscope, respectively. Coordinate transformation is then performed using the pixel density coefficient. , ,in is the pixel density coefficient, with a value of 0.015 cm / pixel, and u and v represent the original coordinate positions of the pixels in the original image. Inverse displacement correction is performed on each pixel of the original image to eliminate geometric deformation and motion blur caused by hand shake, resulting in the compensated image.

[0026] Optionally, in some embodiments, adaptive noise reduction processing can also be performed on the compensated image. The gray-level variance of each local region of the image is calculated. When the regional variance is greater than a preset threshold (e.g.) When the variance is greater than 50, it is identified as a high-detail region, and a large-window bilateral filter (e.g., 7×7, spatial Gaussian standard deviation 2.0, gray-level similarity standard deviation 30) is used to smooth noise while preserving edge geometric features; when the region variance is small (e.g., If the area is ≤50, it is considered a flat region. A small window Gaussian filter (e.g., 3×3) is used to suppress high-frequency noise while preserving edge geometric features.

[0027] Optionally, in some embodiments, the denoised image can be lightweight sampled and its resolution standardized based on the real-time computing capabilities of the mobile device. A bilinear interpolation algorithm is used to normalize the image to a preset resolution (e.g., 640×480 pixels), thereby compressing the original approximately 2MB of data to 100KB-400KB, a compression ratio of approximately 1 / 50-1 / 10, generating standardized corrected image data. This corrected image data not only eliminates geometric distortion caused by imaging jitter but also effectively preserves key geometric features such as edges and corners through an adaptive denoising strategy, providing high-quality input with data size and quality adapted to the resource constraints of the mobile device for subsequent lightweight feature extraction steps. During the correction process, the fusion of motion sensor data directly participates in geometric correction in the compensation stage, while the denoising and sampling stages further optimize the image data quality and volume based on the compensation.

[0028] In some embodiments of this application, a motion blur kernel can be generated based on motion sensing data. This motion blur kernel is then used to deblur the image data, thereby obtaining corrected image data. In one implementation, motion sensing data, including angular velocity and acceleration information, can be acquired in real time from an inertial measurement unit (such as a three-axis gyroscope and a three-axis accelerometer). An integral operation is performed to obtain the pose change sequence of the imaging device in three-dimensional space during the exposure time. Based on this pose change sequence, a two-dimensional motion trajectory projected onto the image sensor plane is estimated. According to this motion trajectory, a point spread function estimation algorithm is used to generate a two-dimensional motion blur kernel corresponding to the motion trajectory. The size of this motion blur kernel corresponds to the pixel displacement range of the trajectory, and the pixel values ​​within the blur kernel are normalized according to the relative time elapsed by the trajectory, thereby accurately characterizing the distribution characteristics of motion blur. Using the aforementioned motion blur kernel as a priori for deconvolution, the image data is deblurred. For example, image data is divided into several image blocks. For each image block, Wiener filtering and deconvolution are performed in the frequency domain based on Fast Fourier Transform. The spectrum of the image block is divided by the spectrum of the motion blur kernel, and a noise suppression term is introduced to obtain the deblurred spectrum. The spatial domain corrected image block is then recovered through Inverse Fourier Transform. By stitching together all the corrected image blocks, the corrected image data is obtained. By directly utilizing motion sensor data to generate a physically constrained blur kernel, image degradation caused by shaking or movement during shooting can be effectively restored, improving the sharpness and signal-to-noise ratio of the corrected image data.

[0029] Step 103: Extract features from the corrected image data using a lightweight convolutional neural network, and adaptively adjust the computational cost of feature extraction based on the device's operating status during the extraction process to obtain the geometric structure feature set output by the lightweight convolutional neural network.

[0030] In some embodiments of this application, the lightweight convolutional neural network uses an improved MobileNetV4 as its backbone, replacing all its standard convolutional layers with depthwise separable convolutions, significantly reducing the total number of parameters (e.g., from 3.2M to 0.4M), and introducing an optimized Swish activation function to reduce non-linear computational complexity, outputting a basic feature map. The Swish activation function can be expressed as... ,in , The sigmoid activation function is used, and β is a learnable parameter. During the basic feature extraction process, the real-time computing power metrics of the mobile device are acquired simultaneously, such as the current GFLOPS value of the GPU. The system dynamically switches the attention calculation mode based on the real-time computing power: when the GPU computing power is greater than or equal to the first threshold (e.g., 800 GFLOPS), a global attention mechanism is adopted to perform global context modeling on the entire image feature map to ensure high perception accuracy; when the GPU computing power is between the second and first thresholds (e.g., 400 GFLOPS to 800 GFLOPS), the Region Proposal Network (RPN) is triggered to generate a preset number (e.g., 100 to 200) of target candidate regions, and local attention is calculated only for the aforementioned target candidate regions to reduce the amount of computation while ensuring the perception effect of key regions; when the GPU computing power is lower than the second threshold (e.g., 400 GFLOPS), channel attention weight allocation is performed, assigning corresponding weight coefficients (e.g., 0.1 to 1.0) to different feature channels according to their importance to minimize computational overhead. After dynamic attention adjustment, the feature map is filtered for geometrically relevant features such as edges, corners, and contours. For example, Canny edge detection and SIFT corner detection algorithms can be used, combined with a dynamic confidence threshold (e.g., 0.7) to remove redundant feature points with low confidence, outputting a structured geometric feature set containing multiple feature points. Each feature point carries three-dimensional spatial coordinates. and normal vector information This provides high-quality sparse geometric features for subsequent geometric enhancement modules to perform structural completion and fusion processing. Through the above process, under the condition of limited mobile resources, a lightweight network is used to reduce the basic computational burden, and a dynamic attention mechanism that is aware of computing power is used to achieve an adaptive balance between feature extraction accuracy and real-time performance, thereby obtaining a structured geometric feature set that is adapted to the performance of different devices.

[0031] In some embodiments of this application, device operating status information, including processor utilization and / or battery level, can also be obtained. Based on this information, the currently available computing overhead level is determined. A feature extraction algorithm matching the computing overhead level is selected to extract features from the corrected image data, resulting in a geometric structure feature set. In one implementation, the current operating status information of the executing device can be obtained, for example, on a mobile terminal such as a smartphone or tablet, by reading the current CPU utilization percentage and the remaining battery percentage through a system API. Assume the CPU utilization is 45% and the battery level is 70%. Based on the device operating status information, the currently available computing overhead level is determined. The computing overhead level can be pre-divided into three or more levels: high, medium, and low, each corresponding to a different computing resource budget. For example, when the CPU utilization is below 30% and the battery level is above 50%, it is determined to be a high overhead level; when the CPU utilization is between 30% and 70% and the battery level is between 20% and 50%, it is determined to be a medium overhead level; otherwise, it is a low overhead level. In this example, CPU usage of 45% and battery usage of 70% may fall into the medium overhead level or be determined according to specific mapping rules. Select a feature extraction algorithm that matches the computational overhead level. For high overhead levels, choose algorithms with high computational complexity and accuracy, such as deep learning-based convolutional neural networks for extracting geometric features; for medium overhead levels, choose traditional algorithms such as Scale Invariant Feature Transform (SIFT) or Speed-Up Robust Feature Transform (SURF); for low overhead levels, choose corner detection algorithms with minimal computational cost, such as FAST or ORB. Execute the selected algorithm on the corrected image data to obtain a geometric feature set. This feature set can contain feature maps at multiple scales, with each feature point carrying spatial coordinates, a normal vector, and local structural context information.

[0032] In some embodiments of this application, the processor idle capacity can be determined based on the processor utilization rate in the device operating status information; the remaining battery life ratio can be determined based on the battery level in the device operating status information; a weighted comprehensive resource index is calculated based on the processor idle capacity and the remaining battery life ratio, and then mapped to a preset computing overhead level based on the weighted comprehensive resource index. When determining the computing overhead level, it is not simply segmented, but rather a more refined adjustment is achieved through quantitative indicators. For example, the processor idle capacity can be obtained by multiplying (100% - processor utilization rate) by the total processor computing power (e.g., clock speed), or simplified to (100% - CPU utilization rate) as the idle ratio. The remaining battery life ratio can be directly obtained by dividing the battery level percentage by 100. The weighted comprehensive resource index is calculated according to preset weights, for example, using the formula: Index = α (Processor idle capacity) + β (Battery remaining range percentage), where α and β are weighting factors that can be adjusted according to actual needs. Typically, α=0.6 and β=0.4 to emphasize computing power. The calculated Index is a value between 0 and 1. The weighted comprehensive resource index is mapped to a computing overhead level, for example, Index ≥ 0.7 is high, 0.3 ≤ Index < 0.7 is medium, and < 0.3 is low. The feature extraction algorithm is selected according to the mapping level. Therefore, even when the battery is fully charged but the CPU is extremely busy, a high overhead level will not be mistakenly selected, thus avoiding system lag and achieving adaptive adjustment.

[0033] Step 104: Based on the preset geometric constraint rules, perform structural completion and fusion processing on the geometric structure feature set to generate a geometric structure model.

[0034] In some embodiments of this application, adaptive multi-scale fusion can be performed on the input features. The Laplacian variance of the feature maps at each scale is calculated as a sharpness index. Higher fusion weights (e.g., 0.6) are assigned to detail layer features with lower sharpness, while lower fusion weights (e.g., 0.4) are assigned to global layer features with higher sharpness. A weighted summation formula is then used. A set of multi-scale fused feature maps integrating local details and global semantics is generated to resolve conflicts and redundancy between features at different scales. Structural missing detection and completion are performed based on a pre-built set of geometric constraint rules. This rule set can include geometric topological rules describing shape constraints and connectivity relationships between common geometric elements, such as rectangular, circular, and polygonal structures and their edge and corner constraints. Specifically, this can be achieved by calculating the gradient changes of feature response values. And a threshold relative to the mean of the global feature response is set to identify occlusion boundaries. For example, when When it is determined to be an occlusion boundary, among which This represents the mean of the global feature response. Once an occluded region is identified, a completion path is generated based on the corresponding geometric topological rules. For example, for a rectangular structure with three known vertices... Then, the coordinates of the fourth vertex are calculated using the rule that opposite sides are parallel and equal. , This process fills in the geometric structure information of the occluded region, resulting in the incomplete features. Furthermore, the parameter accuracy of the incomplete geometry is optimized: using the coordinates of the structural vertices in the incomplete features as optimization variables, a least-squares objective function is constructed. ,in , , To fit the coordinates, , , Using the original coordinates, for example, minimizing the sum of squares of the differences between the fitted coordinates and the original coordinates, the geometric parameters such as the vertex coordinates, side lengths, and included angles of the structure are adjusted through iterative solutions (e.g., 50 iterations or reaching the convergence threshold 1e-5) to reduce the fitting error. Finally, a target geometric structure model containing parameters such as size, spatial coordinates, included angles, and confidence is generated and passed to the subsequent output stage.

[0035] In some embodiments of this application, missing geometric structure regions in a geometric structure feature set can be identified; based on geometric constraint rules, structural deduction is performed on the missing geometric structure regions to generate a complete geometric structure; the complete geometric structure is then fused with the initial geometric structure in the geometric structure feature set to generate a geometric structure model. Specifically, for the complete geometric structure, boundary geometric features adjacent to the missing geometric structure regions can be extracted from the geometric structure feature set; based on geometric constraint rules, the boundary geometric features are extended and constrained to optimize the geometric topology of the missing region, and the deduced geometric topology is converted into a complete geometric structure.

[0036] In one implementation, missing regions can be automatically identified by analyzing the connectivity and continuity of each feature element (such as vertices, edges, faces, or combinations thereof) within a geometric feature set. For example, for a building facade feature set, if there are obvious discontinuities or voids in the point cloud density or feature line segments that do not meet the preset complete geometric primitive closure conditions, the discontinuous area is marked as a missing geometric structure region. This identification process can also be combined with topological constraints of the feature set, such as loop closure rules and surface boundary consistency, to distinguish between actual missing regions and designed openings. Boundary geometric features adjacent to the missing geometric structure region are extracted from the geometric feature set, such as curve segments at the edge of the missing region, normal vectors and curvatures of the feature point cloud, and equations of adjacent planes or surfaces. Based on preset geometric constraint rules, the extracted boundary geometric features are extended and constrained for optimization to infer the geometric topology of the missing region. These geometric constraint rules may include continuity constraints (such as G0 positional continuity, G1 tangent continuity, or G2 curvature continuity), symmetry constraints, parallel / perpendicular constraints, proportional constraints, and consistency constraints of feature space distribution. For example, for a missing window bottom edge structure, if the extracted adjacent boundary features show that the left and right window frames are symmetrically distributed and the top and bottom edges are parallel, then the constraint rules can drive the boundary curves to extend along the tangent direction while maintaining symmetry and parallelism. Simultaneously, optimization is performed by minimizing the curvature variation, thereby inferring the most likely geometric contour topology of the missing region. The inferred geometric topology is then converted into a complete geometric structure. Conventional geometric modeling techniques such as surface fitting, lofting, or sweeping can be used to transform the boundary curves and topological skeleton into continuous geometric surfaces or solids, generating the completed partial structure. The completed geometric structure is then stitched together with the surrounding original geometric structure, and stitching edge optimization, overlapping surface merging, tolerance adjustment, and overall smoothing are applied to ensure that the fused region is geometrically and topologically continuous and consistent, without redundant gaps or overlaps, forming a geometric structure model.

[0037] In some embodiments of this application, after generating the geometric structure model, it can be output in various forms such as visual annotation and structured parameter data, and can receive user interaction feedback to continuously optimize the perception accuracy. For example, the geometric structure model can be encapsulated into a preset data format, and the encapsulated geometric structure model can be sent to the augmented reality application through an interface. The core parameters of the geometric structure model include target identifier, structure type (such as rectangle, circle or polygon), spatial coordinates, size, included angle, and confidence level (0~1).

[0038] In one implementation, AR outlines (such as 2-pixel-wide red lines) can be overlaid on the mobile camera preview screen, with dimension information annotated. A geometric feature confidence heatmap based on Jet color mapping is generated, with confidence ranging from 0 to 1 corresponding to a blue-to-red gradient, visually demonstrating the reliability of the perception results. For complex geometric structures, a lightweight 3D preview model with no more than 5000 triangular faces can be generated, ensuring a rendering frame rate of no less than 30fps to provide a smooth interactive viewing experience on resource-constrained mobile devices. The geometric structure model parameters are encapsulated as structured data in JSON format, which may include fields such as target identifier, structure type, length, width, included angle, center coordinates, measurement error range, and timestamp. This data is synchronously output to the AR SDK or third-party measurement applications via a RESTful API interface, achieving seamless integration with external systems. Examples of core fields are as follows: { "target_id":"obj_001", / / Target ID (unique identifier) "structure_type":"rectangle", / / Structure type (rectangle / circle / polygon) "parameters":{ "length":"120.5cm", / / Length (unit: cm) "width":"60.2cm", / / Width (unit: cm) "angle":"0.3°", / / Angle with the horizontal direction (unit: °) "center_xyz":["150.3mm","200.1mm","180.0mm"], / / Center coordinates (unit: mm) "error_range":"±1mm" / / Measurement error range }, "confidence": 0.92, / / confidence level "timestamp":"2025xxxx_xxxxxx" / / Timestamp (format: year month day_hour minute second) } In addition, some embodiments may provide a touch screen interaction entry, allowing users to manually adjust the geometric contour by dragging and dropping annotation points or smearing correction areas, record the user's correction data and store it in a local database. When the accumulated effective correction data reaches a preset number (e.g., 10), the geometric prior knowledge base is updated in batches. The updated knowledge base is automatically called in subsequent perception tasks, thereby forming a closed loop of perception-feedback-optimization, continuously improving the accuracy and robustness of subsequent perception models.

[0039] By implementing the embodiments of this application, image and motion sensing data are simultaneously acquired on the mobile device and correlated over time. Image correction processing based on motion sensing data eliminates jitter interference. Adaptive noise reduction and resolution standardization significantly reduce data volume while preserving geometric edges. By using a lightweight network and adaptively switching computing modes according to the device's real-time operating status, both the accuracy and real-time performance of feature extraction can be balanced under resource-constrained conditions. Multi-scale fusion and structural completion based on preset geometric constraint rules can repair structural breaks caused by occlusion, texture loss, etc., generating a complete and highly accurate target geometric structure model. This application can, to a certain extent, solve the problems of mobile geometric structure perception being susceptible to jitter interference, requiring high computing power, and incomplete perception in complex scenarios such as occlusion. It achieves highly robust and high-precision real-time geometric structure perception and modeling on resource-constrained mobile terminals.

[0040] Figure 2 This is a schematic diagram of a mobile terminal geometric structure model building device provided in an embodiment of this application. Figure 2 As shown, the mobile terminal geometric structure model building device may include: an acquisition module 201, a processing module 202, an extraction module 203, and a generation module 204.

[0041] The acquisition module 201 is used to acquire image data collected by the mobile terminal and motion sensing data associated with the image data.

[0042] The processing module 202 is used to perform correction processing on image data based on motion sensing data to obtain corrected image data.

[0043] The extraction module 203 is used to extract features from the corrected image data using a lightweight convolutional neural network, and adaptively adjusts the computational cost of feature extraction based on the device's operating status during the extraction process to obtain the geometric structure feature set output by the lightweight convolutional neural network.

[0044] The generation module 204 is used to perform structural completion and fusion processing on the geometric structure feature set based on preset geometric constraint rules to generate a geometric structure model.

[0045] In some embodiments of this application, the processing module 202 is specifically used to: generate a motion blur kernel based on motion sensing data; and use the motion blur kernel to deblur the image data to obtain corrected image data.

[0046] In some embodiments of this application, the extraction module 203 is specifically used to: obtain device operating status information, including processor utilization and / or battery power; determine the currently available computing overhead level based on the device operating status information; select a feature extraction algorithm that matches the computing overhead level, perform feature extraction on the corrected image data, and obtain a geometric structure feature set.

[0047] In some embodiments of this application, the extraction module 203 is specifically used to: determine the processor idle capacity based on the processor utilization rate in the device operating status information; determine the remaining battery life ratio based on the battery power in the device operating status information; calculate a weighted comprehensive resource index based on the processor idle capacity and the remaining battery life ratio, and map the weighted comprehensive resource index to a preset computing overhead level.

[0048] In some embodiments of this application, the generation module 204 is specifically used to: determine the missing geometric structure region in the geometric structure feature set; perform structural calculation on the missing geometric structure region according to geometric constraint rules to generate a complete geometric structure; and fuse the complete geometric structure with the initial geometric structure in the geometric structure feature set to generate a geometric structure model.

[0049] In some embodiments of this application, the generation module 204 is specifically used to: extract boundary geometric features adjacent to the missing geometric structure region from the geometric structure feature set; extend and optimize the boundary geometric features based on geometric constraint rules to deduce the geometric topology of the missing region; and convert the deduce geometric topology into a complete geometric structure.

[0050] In some embodiments of this application, the acquisition module 201 is specifically used to: acquire image data through an image acquisition device on a mobile terminal, and synchronously acquire motion sensing data through an inertial measurement unit; and align the image data and motion sensing data with timestamps to obtain image data and motion sensing data associated with the image data.

[0051] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0052] To implement the above embodiments, this application also proposes an electronic device, including: a processor and a memory communicatively connected to the processor; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0053] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0054] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0055] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0056] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0057] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0058] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0059] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0060] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0061] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0062] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for constructing a geometric structure model for a mobile terminal, characterized in that, include: Acquire image data collected by the mobile device and motion sensing data associated with the image data; The image data is corrected based on the motion sensing data to obtain corrected image data. The corrected image data is feature extracted using a lightweight convolutional neural network, and the computational cost of feature extraction is adaptively adjusted based on the device's operating status during the extraction process to obtain the geometric structure feature set output by the lightweight convolutional neural network. Based on preset geometric constraint rules, the geometric structure feature set is subjected to structural completion and fusion processing to generate a geometric structure model.

2. The method according to claim 1, characterized in that, The step of correcting the image data based on the motion sensing data to obtain corrected image data includes: A motion fuzzy kernel is generated based on the motion sensing data; The image data is deblurred using the motion blur kernel to obtain corrected image data.

3. The method according to claim 1, characterized in that, During the extraction process, the computational cost of feature extraction is adaptively adjusted based on the device's operating status to obtain the geometric structure feature set output by the lightweight convolutional neural network, including: Obtain device operating status information, which includes processor utilization and / or battery level; Determine the currently available computing overhead level based on the device operating status information; Select a feature extraction algorithm that matches the computational overhead level, extract features from the corrected image data, and obtain the geometric structure feature set.

4. The method according to claim 3, characterized in that, The step of determining the currently available computing overhead level based on the device operating status information includes: Based on the processor utilization rate in the device operating status information, determine the processor idle capacity; Based on the battery level in the device's operating status information, determine the remaining battery range percentage; Based on the processor's idle capacity and the remaining battery life, a weighted comprehensive resource index is calculated, and the weighted comprehensive resource index is mapped to a preset computing overhead level.

5. The method according to claim 1, characterized in that, The process of performing structural completion and fusion on the geometric feature set based on preset geometric constraint rules to generate a geometric structure model includes: Identify the missing geometric structure regions in the set of geometric structure features; Based on the geometric constraint rules, structural calculations are performed on the missing geometric structure region to generate a complete geometric structure; The completed geometry is fused with the initial geometry in the set of geometric features to generate the geometric model.

6. The method according to claim 5, characterized in that, The step of performing structural deduction on the missing geometric structure region according to the geometric constraint rules to generate a complete geometric structure includes: Extract the boundary geometric features adjacent to the missing geometric structure region from the set of geometric structure features; Based on the geometric constraint rules, the boundary geometric features are extended and constrained to optimize them in order to deduce the geometric topology of the missing region; The calculated geometric topology is converted into the completed geometric structure.

7. The method according to claim 1, characterized in that, The acquisition of image data collected by the mobile terminal and motion sensing data associated with the image data includes: Image data is acquired through the image acquisition device on the mobile terminal, and motion sensing data is acquired synchronously through the inertial measurement unit; The image data and the motion sensing data are time-stamped to obtain the image data and the motion sensing data associated with the image data.

8. A device for constructing a geometric structure model for a mobile terminal, characterized in that, include: The acquisition module is used to acquire image data collected by the mobile terminal and motion sensing data associated with the image data; The processing module is used to perform correction processing on the image data based on the motion sensing data to obtain corrected image data; The extraction module is used to extract features from the corrected image data using a lightweight convolutional neural network, and adaptively adjust the computational cost of feature extraction based on the device operating status during the extraction process to obtain the geometric structure feature set output by the lightweight convolutional neural network. The generation module is used to perform structural completion and fusion processing on the geometric feature set based on preset geometric constraint rules to generate a geometric structure model.

9. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images