A vector diagram generation method and device, a storage medium and an electronic device

Abstract

The application discloses a vector graph generation method and device, a storage medium and an electronic device. The method comprises the following steps: based on collected target video data, an image data set with semantic information is constructed; three-dimensional reconstruction is performed according to images and semantic information in the image data set, and three-dimensional point cloud data with semantic information is obtained; a ground surface is fitted according to ground point cloud data extracted from the three-dimensional point cloud data, and target semantic alignment point cloud data is obtained; a semantic density map corresponding to semantic category point cloud in the target semantic alignment point cloud data is constructed, and a vector graph with semantic perception corresponding to the target video data is generated, so that the integrity and geometric precision of key element reconstruction in a vector graph generation application scenario are improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a method and apparatus for generating vector graphics. This application also relates to a computer storage medium and an electronic device. Background Technology

[0002] With the deepening of digital transformation, the operation and management of indoor spaces such as large commercial complexes, supermarkets, exhibition halls, and 4S stores are shifting from traditional experience-driven to data-driven intelligent operation models. Advanced applications such as customer flow analysis, merchandise display optimization, and regional heat mapping have become key means to enhance space value and operational efficiency.

[0003] However, the effective implementation of these smart applications highly depends on the accuracy and completeness of the underlying spatial data. Currently, the industry generally faces the following technical bottlenecks: 1. The "Geometric Island" Dilemma of Traditional CAD Drawings Traditional computer-aided design drawings, while accurately recording the geometric information of buildings (such as walls, column grids, doors, and windows), typically consist of underlying data models composed only of basic geometric elements such as points, lines, and surfaces. These drawings lack the ability to interpret the meaning of the related elements behind these geometric elements, making it impossible for computers to distinguish whether a line represents a "load-bearing wall" or a "movable shelf." Drawings containing only geometric information and lacking semantic information are ill-suited to supporting complex higher-level analysis and decision-making algorithms.

[0004] 2. The disconnect between spatial data and application scenario data In current smart business solutions, operational data (such as customer flow patterns and sales figures) is often stored independently of spatial data. Due to the lack of semantically meaningful electronic base maps, customer flow heatmaps can only be coarsely overlaid on pixel-based images, failing to pinpoint the exact "shelf" or "stacking" location. This results in "blind spots" in data analysis; operations personnel may know that a certain area is crowded, but they cannot accurately locate the corresponding specific display area or promotional point, leaving refined operational measures such as display optimization without reliable data support.

[0005] 3. Lack of dynamic element management Commercial spaces are not static. The location and form of dynamic elements such as promotional displays, temporary showcases, and seasonal displays change frequently. Traditional static CAD drawings cannot record the semantic information (such as "this is a promotional display" or "this is a refrigerated display") and their temporal attributes of these dynamic elements, resulting in a disconnect between digital models and physical reality, and making it impossible to achieve real-time digital mirroring.

[0006] Therefore, a new CAD drawing construction technology is urgently needed. This technology should not only possess high-precision geometric measurement capabilities but also endow drawing elements with complete semantic tags. By hierarchically and categorizing building structures (walls / columns) and application scenario-related elements (shelves / stalls / display cases) in a structured manner, a precise mapping from the physical world to the digital world can be established. This provides the upper-level intelligent operation system with high-quality basic data support that can see both the "skeleton" (building) and the "flesh" (business). Summary of the Invention

[0007] This application provides a method for generating vector graphics to solve the problem that vector graphics in the prior art lack semantic information and cannot be used by downstream applications.

[0008] This application provides a method for generating vector graphics, including: Based on the collected target video data, an image dataset with semantic information is constructed. Three-dimensional reconstruction is performed based on the images and semantic information in the image dataset to obtain three-dimensional point cloud data with semantic information. The ground is fitted with ground point cloud data extracted from the three-dimensional point cloud data to obtain target semantically aligned point cloud data; Based on the semantic density map corresponding to the semantic category point cloud in the constructed point cloud data aligned with the target semantics, a semantically aware vector image corresponding to the target video data is generated.

[0009] In some embodiments, constructing an image dataset with semantic information based on the acquired target video data includes: Collect panoramic video data of the target scene as the target video data; Extract keyframe images from the target video data to obtain a keyframe image set; The keyframe images in the keyframe image set are segmented to obtain a pinhole view set; Semantic segmentation is performed on the pinhole views in the pinhole view set to obtain an image dataset with semantic information.

[0010] In some embodiments, segmenting the keyframe images in the keyframe image set to obtain a pinhole view set includes: The segmentation operation is defined based on the virtual camera configuration parameter set; According to the segmentation operation, the keyframe images in the keyframe image set are converted from equidistant cylindrical projection into multiple forward pinhole views; The plurality of forward pinhole views are defined as the pinhole view set.

[0011] In some embodiments, the step of semantically segmenting the pinholes in the pinhole view set to obtain an image dataset with semantic information includes: A predefined set of image semantic categories; The image dataset with semantic information is constructed based on the pinhole views in the pinhole view set and the image semantic category set.

[0012] In some embodiments, the step of performing 3D reconstruction based on images and semantic information in the image dataset to obtain 3D point cloud data with semantic information includes: The input data for the semantic segmentation model is defined based on the images in the image dataset and the semantic information corresponding to the images. The image and the semantic information are used as a data pair and input into the geometric feature extraction submodule and the semantic feature extraction submodule of the 3D reconstruction network model to extract geometric features and semantic features corresponding to the spatial location of the geometric features, respectively. The geometric features and the semantic features are identified as a multimodal feature pair; Semantic-gated feature matching is performed on the multimodal feature pairs to generate a set of matching feature pairs; The feature pairs in the matching feature pair set are reconstructed using a 3D reconstruction method to obtain the 3D point cloud data with semantic information.

[0013] In some embodiments, the step of performing 3D reconstruction on the feature pairs in the matching feature pair set to obtain the 3D point cloud data with semantic information includes: Based on the set of matching feature pairs, a 3D reconstruction method is performed to obtain an initial sparse point cloud and camera pose. Based on the initial sparse point cloud, the camera pose, and the defined weights, semantic-aware joint optimization is performed on the initial sparse point cloud and the camera pose to obtain the 3D point cloud data with semantic information.

[0014] In some embodiments, fitting the ground based on the ground point cloud data extracted from the three-dimensional point cloud data to obtain target semantic point cloud data includes: The three-dimensional point cloud data is downsampled to obtain downsampled three-dimensional point cloud data; Ground point cloud datasets are extracted from the downsampled 3D point cloud data using semantic information. Select planes that meet the requirements from the ground point cloud dataset to perform ground fitting and generate the fitted ground equation; The three-dimensional point cloud data is corrected according to the fitted ground equation to generate the target semantic point cloud data.

[0015] In some embodiments, selecting a plane that meets the requirements from the ground point cloud dataset for ground fitting and generating a fitted ground equation includes: The plane is determined based on non-collinear points randomly selected from the ground point cloud dataset; Count the number of interior points in the ground point cloud dataset whose distance from the plane is less than or equal to a preset threshold; Determine whether the number of interior points is greater than or equal to a preset minimum interior point threshold; If so, the plane is determined as a valid candidate plane model; During the process of repeating the above steps according to the number of iterations, if the number of interior points of the current valid candidate plane model is greater than or equal to the historical maximum number of interior points, then the valid candidate plane model is updated. The fitted ground equation is generated based on the updated valid candidate plane model.

[0016] In some embodiments, the step of performing point cloud correction on the three-dimensional point cloud data according to the fitted ground equation to generate the target semantically aligned point cloud data includes: Based on the fitted ground equation normal vector and the reference plane normal vector, the rotation axis and rotation angle are obtained; Based on the rotation axis and the rotation angle, obtain the rotation matrix; The fitted ground equation and the downsampled 3D point cloud data are rotated and / or translated according to the rotation matrix to generate the target semantically aligned point cloud data.

[0017] In some embodiments, generating a semantically aware vector map corresponding to the target video data based on the semantic density map corresponding to the semantic category point cloud in the constructed point cloud data aligned with the target semantics includes: Based on the target semantic alignment point cloud data, construct a semantic density map corresponding to the semantic category point cloud; Based on the semantic density map, extract the intra-class boundaries to obtain the intra-class boundary set; The vector diagram is generated based on the intra-class boundaries in the intra-class boundary set.

[0018] In some embodiments, constructing a semantic density map corresponding to the semantic category point cloud based on the target semantic aligned point cloud data includes: According to the semantic category of the semantic point cloud in the target semantic alignment point cloud data, a density map is generated by projecting it onto the plane of the fitted ground. The density map is post-processed to obtain the semantic density map.

[0019] In some embodiments, extracting intra-class boundaries to obtain an intra-class boundary set based on the semantic density map includes: Edge enhancement and line segment detection are performed on the semantic density map to obtain an initial set of line segments; The dominant structure is obtained by fitting dominant lines to the endpoints of the line segments in the initial set of line segments. Construct a planar line diagram based on the dominant structure; The polygons extracted from the planar straight line graph are used as the intra-class boundary set.

[0020] In some embodiments, it also includes: Based on the intra-class boundary set, conflict detection is performed on the boundaries between different categories to determine whether the boundaries need to be corrected. If so, the correction is performed, and the corrected boundary is determined as the boundary set within the target class.

[0021] In some embodiments, the step of performing conflict detection on the boundaries between different categories based on the intra-class boundary set, and determining whether to correct the boundaries, includes: Penetration conflict detection and / or overhang conflict detection are performed on the boundaries between the different categories to determine whether a conflict exists between the different categories and the type of conflict. If so, the boundary is corrected according to the conflict type.

[0022] In some embodiments, the step of performing penetration conflict detection and / or overhang conflict detection between the different categories to determine whether a conflict exists and the type of conflict between the different categories includes: Construct a buffer for one of any two different classes of polygons; Determine whether there is an intersection between the buffer corresponding to the polygon of the stated category and the polygon of another category; If so, then it is determined that there is a conflict between the two different types of polygons, and the conflict type is a penetration conflict type; And / or, Determine whether the height of the lowest point in the semantic point cloud subset corresponding to any category of polygon in the intra-class boundary set is greater than the sum of the ground height and the height tolerance threshold. If so, it is determined that the polygons corresponding to the semantic point cloud subset have a conflict, and the conflict type is a dangling conflict type.

[0023] In some embodiments, the conflict correction of the boundary according to the conflict type includes: When the conflict type is a penetration conflict type, one of the penetration conflict pairs is set as a movable point and the other is set as a fixed point, and the signed distance from the movable point to the buffer is calculated; Calculate the minimum translation correction based on the signed distance; The boundary of the polygon is corrected according to the minimum translation correction amount; And / or, When the conflict type is a dangling conflict type, the polygons with dangling conflicts and the elements related to the polygons are deleted from the class boundary set.

[0024] This application also provides a vector graphic generation apparatus, comprising: The building unit is used to construct an image dataset with semantic information based on the acquired target video data; The reconstruction unit is used to perform three-dimensional reconstruction based on the images and semantic information in the image dataset to obtain three-dimensional point cloud data with semantic information. The fitting unit is used to fit the ground based on the ground point cloud data extracted from the three-dimensional point cloud data to obtain the target semantic point cloud data. The generation unit is used to generate a semantically aware vector image corresponding to the target video data based on the constructed semantic density map corresponding to the semantic category point cloud in the target semantic point cloud data.

[0025] This application also provides a computer storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform the steps in the vector graphic generation method described above.

[0026] This application also provides an electronic device, including: processor; The memory is used to store programs that process data generated by the electronic device. When the program is read and executed by the processor, it performs the steps in the vector graphic generation method described above.

[0027] Compared with the prior art, this application has the following advantages: This application provides a method for generating vector graphics. On one hand, it constructs an image dataset with semantic information. During the 3D reconstruction process based on this dataset, a semantically guided incremental approach is used, resulting in 3D point cloud data with semantic information. Further, the aligned point cloud data obtained by fitting the ground also contains semantic information. Semantic information is also embedded during the density map construction process, ultimately generating a semantically aware vector graphic. This process improves the completeness and geometric accuracy of key element reconstruction in vector graphic generation applications, avoiding the problem of focusing only on building structures while ignoring key elements such as shelves and pushers. It also addresses the shortcomings of sparse and small structures being easily missed, and sensitivity to point cloud density. On the other hand, by introducing a semantic segmentation model, it can accurately segment any specified task object without retraining, significantly improving semantic generalization ability. Furthermore, it eliminates the need for manual annotation, achieving fully automatic output of vector graphics. On the other hand, by introducing a category-adaptive uncertainty-weighted joint optimization strategy in the semantic-aware 3D reconstruction process, key elements such as shelves and display cases are dynamically assigned higher reconstruction weights during global joint optimization. This allows key elements to achieve high-confidence 3D localization even under low-texture and weak observation conditions, improving the problem of reconstruction failure or deformation in sparse regions by existing uniform weighting methods. Finally, in the generation of the semantic density map, spatial relationship correction of different category boundaries is implemented. This avoids problems such as high-density wall structures obscuring sparse shelf outlines. Furthermore, through penetration conflict detection and minimum translation correction, it ensures that the output CAD drawing meets geometric prior constraints such as objects not penetrating each other and service elements or object facilities contacting the ground, improving the accuracy and usability of the generated CAD drawing. Attached Figure Description

[0028] Figure 1 This is a flowchart of a vector graphic generation method provided in this application.

[0029] Figure 2 This is a flowchart of an example of constructing an image dataset with semantic information.

[0030] Figure 3 This is a schematic diagram of semantic segmentation in an embodiment of obtaining 3D point cloud data with semantic information.

[0031] Figure 4 This is a flowchart of an embodiment for obtaining 3D point cloud data with semantic information.

[0032] Figure 5 This is a schematic diagram of a 3D reconstruction network structure in an embodiment of obtaining 3D point cloud data with semantic information.

[0033] Figure 6 This is a flowchart of the fitted ground implementation example.

[0034] Figure 7 This is a schematic diagram of the fitting of the ground embodiment.

[0035] Figure 8 This is a schematic diagram of point cloud verticalization in the implementation of the fitted ground.

[0036] Figure 9 This is a flowchart of an example of generating semantically aware vector graphics.

[0037] Figure 10 This is a schematic diagram of the structure of a vector graphic generation device provided in this application.

[0038] Figure 11 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0039] Many specific details are set forth in the following description to provide a full understanding of this application. However, this application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this application; therefore, this application is not limited to the specific embodiments disclosed below.

[0040] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The descriptive terms used in this application and the appended claims, such as "a," "first," and "second," are not intended to limit quantity or sequence, but rather to distinguish information of the same type from one another.

[0041] As described in the background section above, the inventive concept of this application originates from the application scenario of large-scale indoor scene digitization, where accurate and complete CAD floor plans are the fundamental support for realizing intelligent operations (such as circulation analysis, display optimization, and customer flow heat map correlation). Large shopping malls and other commercial spaces not only require architectural geometric information such as walls and room outlines, but also the accurate representation of key business elements such as shelves, display cases, and displays. Therefore, higher requirements are placed on the semantic integrity and geometric accuracy of vector CAD drawings. However, existing technologies for generating indoor vector CAD drawings include: 3D scanning solutions based on LiDAR or depth cameras (such as LiDAR and RGB-D sensors) can obtain high-precision point clouds, but the equipment is expensive and the deployment is complex, making it difficult to quickly scale up in existing stores. SLAM (Simultaneous Localization and Mapping) reconstruction based on multi-view static images: Although ordinary cameras can be used, it usually requires manual setup of shooting paths, control of lighting and occlusion, and the process is cumbersome and not user-friendly for non-professional users. While traditional manual mapping can guarantee semantic accuracy, it is time-consuming, costly, and difficult to iterate. Dynamic acquisition solutions based on panoramic or monocular video have the advantages of flexible deployment and low cost. However, existing methods mostly focus on building structure reconstruction and lack the ability to effectively model sparse and low-texture business elements such as shelves and display cases. Furthermore, they generally do not integrate fine-grained semantic information, making it difficult to directly use the generated CAD drawings for operational analysis.

[0042] Based on the above, this application provides a method for generating vector graphics, such as... Figure 1 As shown, the method includes: Step S101: Based on the acquired target video data, construct an image dataset with semantic information; Step S102: Perform 3D reconstruction based on the images and semantic information in the image dataset to obtain 3D point cloud data with semantic information; Step S103: Fit the ground based on the ground point cloud data extracted from the three-dimensional point cloud data to obtain target semantically aligned point cloud data; Step S104: Generate a semantically aware vector map corresponding to the target video data based on the semantic density map corresponding to the semantic category point cloud in the constructed point cloud data aligned with the target semantics.

[0043] The steps S101 to S104 described above will be described in detail below with reference to specific embodiments.

[0044] Regarding step S101: Based on the acquired target video data, construct an image dataset with semantic information.

[0045] like Figure 2 As shown, the purpose of this step is to construct an image dataset with semantic information based on the target video data. The specific implementation process may include: Step S101-1: Collect panoramic video data of the target scene as the target video data; Step S101-2: Extract keyframe images from the target video data to obtain a keyframe image set; Step S101-3: Segment the keyframe images in the keyframe image set to obtain a pinhole view set; Step S101-4: Perform semantic segmentation on the pinhole images in the pinhole view set to obtain an image dataset with semantic information.

[0046] In this embodiment, the target scene in step S101-1 can be either an indoor scene or an outdoor scene; that is, any scene that meets the requirements of downstream applications can be the target scene in this embodiment. This embodiment specifically uses an indoor scene as an example for illustration.

[0047] The acquisition process can employ a monocular camera to perform 360-degree panoramic acquisition around the scene, obtaining panoramic video data. This ensures visual coverage and avoids the difficulties of multi-camera synchronization. For example, the acquired panoramic target video data can be... ,in Indicates the first An equidistant cylindrical projection image of a frame. Its height and width.

[0048] To eliminate redundant frames in the target video data, this embodiment uses step S101-2 to extract keyframes. For example, feature matching difference is used to extract keyframes, and a keyframe image set is constructed based on the keyframe images. In this embodiment, a keyframe image set can be defined. for: in: This is the ORB (Oriented FAST and Rotated BRIEF) feature descriptor extraction function, which outputs a normalized feature vector. The threshold for motion significance; The minimum time interval; The timestamp of the previous keyframe is used. That is, if the feature difference between the current frame and the previous frame is greater than the motion saliency threshold, and the difference between the timestamp of the current frame and the timestamp of the previous frame is greater than or equal to the minimum time interval, then the current frame is a keyframe.

[0049] The purpose of step S101-3 is to process the keyframe images in the keyframe image set. The specific implementation process of converting an equidistant cylindrical projection image into multiple forward pinhole views can include: Steps S101-31: Define the segmentation operation based on the virtual camera configuration parameter set; wherein, the virtual configuration parameter set may include: camera intrinsic parameters (such as focal length, principal point, etc.), image resolution, field of view, and extrinsic parameters (such as position, orientation, etc.). Define the segmentation operation. for: in, For the first The rotation matrix of a virtual camera relative to the panoramic coordinate system. For normalized focal length, This represents the perspective remapping function based on spherical back projection.

[0050] Step S101-32: According to the segmentation operation, the keyframe images in the keyframe image set are converted from equidistant cylindrical projection into multiple forward pinhole views; Step S101-33: Determine the plurality of forward pinhole views as the pinhole view set. Wherein, the pinhole view set is... , The number of circumferential segments in the panoramic image determines the output pinhole view resolution. .

[0051] The purpose of step S101-4 is to perform semantic segmentation on the pinhole views, that is: to segment the pinhole views (also called pinhole images) in the pinhole view set using an image semantic segmentation model. Semantic segmentation is performed, and the specific implementation process includes: Step S101-41: Predefine the image semantic category set; for example: define the category set as: like Figure 3 As shown, the defined category set includes key object categories within the scene, such as: walls ( ),Pillar( ),ground( ), shelves ( ),tray( Display cases, push-button displays, etc. The above are merely examples; the category set can be dynamically configured according to different scenario requirements. In this embodiment, semantic information can be the semantic labels possessed by pixels in the image, such as: walls, floors, shelves, etc. Of course, the semantic information possessed by pixels differs in different application scenarios.

[0052] Steps S101-42: Construct the image dataset with semantic information based on the pinhole views in the pinhole view set and the image semantic category set. Specifically, this can be achieved by using the pinhole images... and category hints As a semantic segmentation model The input data is processed by a semantic segmentation model, which outputs a semantic mask tensor: in, For category The binary mask can be understood as: Given a two-dimensional image, it labels all pixels in the original image that belong to the m-th category. Finally, an image dataset with semantic information is constructed: The semantic information can be in the form of semantic tags.

[0053] Regarding step S102: Perform 3D reconstruction based on the images and semantic information in the image dataset to obtain 3D point cloud data with semantic information.

[0054] like Figure 4 and Figure 5 As shown, the purpose of this step is to obtain 3D point cloud data with semantic information through 3D reconstruction. The specific implementation process may include: Step S102-1: Define the input data for the semantic segmentation model based on the images in the image dataset and the semantic information corresponding to the images; the image dataset In For the first Zhang RGB original image, This is the semantic mask corresponding to the original RGB image. The total number of categories is predefined. In this embodiment, the image dataset does not require additional depth annotation or camera calibration; the 3D reconstruction process can be driven by the segmentation results of the image semantic segmentation model (steps S101-4) and the original image.

[0055] Step S102-2: Input the image and the semantic information as data pairs into the geometric feature extraction submodule and the semantic feature extraction submodule of the 3D reconstruction network model to extract geometric features and semantic features corresponding to the spatial location of the geometric features, respectively; for example Figure 5 As shown, the image and semantic information As data inputs to a 3D reconstruction network model, the 3D reconstruction network model comprises four modules: a geometric-semantic feature extraction module, a semantically gated feature matching module, an initial structure and pose estimation module, and a semantic-aware joint optimization module. The geometric-semantic feature extraction module includes two branches: a geometric feature extraction submodule and a semantic feature extraction submodule. The geometric feature extraction submodule is used to extract images. Dense geometric features Geometric extraction Algorithms such as SIFT feature descriptors and DINOv2 can be used, but are not limited to. The semantic feature extraction submodule is used to standardize and spatially align the semantic mask, generating semantic identifiers corresponding to the spatial locations of geometric features. ,in One-hot Semantic vectors (or semantic features) are generated by pixel-by-pixel encoding. In this embodiment, the geometric feature extraction submodule and the semantic feature extraction submodule are two parallel submodules, and the image... You can input these two parallel submodules at the same time.

[0056] Step S102-3: Determine the geometric features and semantic features as a multimodal feature pair. .

[0057] Step S102-4: Perform semantically gated feature matching on the multimodal feature pairs to generate a set of matching feature pairs; in this embodiment, during the cross-view matching stage, semantically consistent feature point pairs can be allowed to participate in the matching, for example: in, The similarity threshold for descriptors (ORB feature descriptors) For feature point pairs, It's an inner product operation. In feature matching, the inner product is typically used to measure the similarity between two feature vectors. The larger the inner product, the closer the two features are in the descriptive subspace. It is a point Geometric feature descriptors. Similarly, It is a point Geometric feature descriptor. Point semantic features and points semantic features The similarity in the feature space is greater than This filters out dissimilar feature point pairs.

[0058] In this embodiment, semantic gating feature matching can effectively suppress cross-category mismatches of key elements and improve the matching accuracy of key elements.

[0059] Step S102-5: Perform 3D reconstruction on the feature pairs in the matching feature pair set using a 3D reconstruction method to obtain the 3D point cloud data with semantic information; the specific implementation process of this step includes: Steps S102-51: Perform a 3D reconstruction method based on the matched feature pair set to obtain the initial sparse point cloud and camera pose; that is, based on the... 3D reconstruction method to obtain initial sparse point cloud With camera pose The 3D reconstruction methods include, but are not limited to, open-source algorithms such as COLMAP and ORB-SLAM3. The execution process may also include filtering operations, such as using RANSAC filtering, to estimate the fundamental or essential matrix in epipolar geometry calculations using RANSAC, thereby filtering out matching points (such as interior points) that meet set constraints and eliminating mismatches (such as bad points). Steps S102-52: Based on the initial sparse point cloud, the camera pose, and the defined weights, perform semantic-aware joint optimization on the initial sparse point cloud and the camera pose to obtain the 3D point cloud data with semantic information. In this embodiment, the following formula can be used to achieve semantic awareness joint optimization: in, For image The set of visible points, For the observation coordinates, For projection function, To optimize the objective, min represents the minimization operation. The weights are defined as follows: in, For indicator functions, when Activate enhancement when it belongs to a critical category; For point The variance of triangulation uncertainty in three-dimensional reconstruction; As a regulating factor, The numerically stable term is then output as a semantically labeled 3D point cloud: Regarding step S103: Fit the ground based on the ground point cloud data extracted from the three-dimensional point cloud data to obtain target semantic point cloud data.

[0060] The purpose of step S103 is to obtain target semantic point cloud data by fitting the ground. For example... Figure 6 and Figure 7 As shown, the specific implementation process of this step includes: Step S103-1: Downsample the 3D point cloud data to obtain downsampled 3D point cloud data; the purpose of this step is to reduce storage and computational overhead. For example, the 3D point cloud with semantic information obtained in steps S102-52 above. Then, differential downsampling processing is performed on the 3D point cloud: in, For category a subset of Indicates voxel resolution as The uniform downsampling operation is used. Voxel resolution is a core concept in 3D spatial data processing. Simply put, it defines the size of the smallest cubic unit into which 3D space is divided. This downsampling process can preserve the original density of point clouds for key categories, downsample the background point cloud, and obtain downsampled 3D point cloud data. .

[0061] Step S103-2: Extract the ground point cloud dataset from the downsampled 3D point cloud data using semantic information; this step can extract ground point data related to the ground point cloud using semantic information, which serves as the ground point cloud dataset. .

[0062] Step S103-3: Select planes that meet the requirements from the ground point cloud dataset for ground fitting, and generate the fitted ground equation; to avoid non-ground interference, this embodiment can detect the ground point cloud dataset. The ground is fitted using the largest plane in the mean, and the specific implementation process includes: Step S103-31: Determine the plane based on non-collinear points randomly selected from the ground point cloud dataset; for example: from the ground point cloud dataset Three non-collinear points are randomly selected from the data. The plane determined by these three points is solved using the least squares method, and the plane parameters of this plane are: Where n represents the normal to the plane, and d represents the constant term of the equation; Step S103-32: Count the number of interior points in the ground point cloud dataset whose distance from the plane is less than or equal to a preset threshold; for example: calculate the number of interior points in the ground point cloud dataset. The perpendicular distance from all points to the plane is statistically less than or equal to a preset threshold. Number of interior points ; Step S103-33: Determine whether the number of interior points is greater than or equal to a preset minimum interior point threshold; for example: determine the number of interior points. Is it greater than or equal to the preset minimum number of interior points threshold? Step S103-34: If yes, then the plane is determined as a valid candidate plane model; Step S103-35: Repeat steps S103-31 to S103-34 according to the number of iterations. If the number of interior points of the current valid candidate plane model is greater than or equal to the historical maximum number of interior points, then update the valid candidate plane model; for example, after reaching a preset number of iterations, update the plane parameters of the valid candidate plane model. ; Step S103-36: Generate the fitted ground equation based on the updated effective candidate plane model, for example: .

[0063] Step S103-4: Perform point cloud correction on the 3D point cloud data according to the fitted ground equation to generate the target semantic point cloud data. In this embodiment, point cloud correction can be achieved through point cloud verticalization and / or centering, such as... Figure 8As shown, the specific implementation process of this step includes: Step S103-41: Obtain the rotation axis and rotation angle based on the fitted ground equation normal vector and the reference plane normal vector; that is: obtain the rotation axis and rotation angle by fitting the ground equation normal vector. With reference plane XY ( Normal vector The cross product is used to obtain the axis of rotation and the angle of rotation, as shown in Formula 10 below: Step S103-42: Obtain the rotation matrix based on the rotation axis and the rotation angle; the rotation matrix can be obtained using the Rodriguez formula, i.e.: in, The cross product matrix; Rodriguez's formula transforms a rotation axis (unit vector) and a rotation angle into a 3×3 rotation matrix. ; Steps S103-43: Rotate and / or translate the fitted ground equation and the downsampled 3D point cloud data according to the rotation matrix to generate the target semantically aligned point cloud data. In this embodiment, rotating and translating the fitted ground equation and point cloud is to ensure that the rotated ground is located in the correct position. The reference plane can be used to define translation vectors. The downsampled 3D point cloud data The corresponding rotation and translation yield the target semantic alignment point cloud data. ,Right now: Regarding step S104: Based on the semantic density map corresponding to the semantic category point cloud in the constructed point cloud data aligned with the target semantics, generate a semantically aware vector map corresponding to the target video data.

[0064] In this embodiment, the purpose of step S104 is to obtain a perceptual vector graphic, such as a semantically perceptual CAD drawing, such as... Figure 9 As shown, the specific implementation process of this step may include: Step S104-1: Based on the target semantic alignment point cloud data, construct a semantic density map corresponding to the semantic category point cloud; Step S104-2: Extract intra-class boundaries based on the semantic density map to obtain an intra-class boundary set; Step S104-3: Generate the vector diagram based on the intra-class boundaries in the intra-class boundary set.

[0065] In this embodiment, to prevent key elements from being overwhelmed by high-density structures such as walls due to the sparse point cloud, independent and resolution-adaptive density maps are generated for different types of point clouds. For example, for different types of point clouds... The density maps are generated by projecting them onto the reference plane (XY plane) respectively, using the following formula: in, To achieve category-adaptive resolution, the unit projection grid size is dynamically set for different semantic categories. For high-density, large-scale structures (such as walls), a coarse resolution can be used. For sparse, small-scale key business elements (such as shelves), a fine resolution approach can be adopted. This is done to preserve the geometric details of the density map. The density map is then then processed... Perform Gaussian smoothing and morphological closing operations: in, For indicator functions, This represents a two-dimensional convolution operation. The standard deviation is Gaussian kernel, Represents the morphological closing operation. It is a structuring element for morphological closing operations.

[0066] Therefore, the specific implementation process of step S104-1 includes: Step S104-11: According to the semantic category of the semantic point cloud in the target semantic alignment point cloud data, project it onto the plane of the fitted ground to generate a density map.

[0067] Step S104-12: Perform density map post-processing on the density map to obtain the semantic density map. In this embodiment, the semantic density map can be understood as a density map containing speech information.

[0068] To distinguish each category, boundary extraction needs to be performed on the density map corresponding to each category. The specific implementation process of step S104-2 includes: Step S104-21: Perform edge enhancement and line segment detection on the semantic density map to obtain an initial line segment set; for example: on the semantic density map The gradient magnitude map is calculated, and an edge map is generated through adaptive threshold binarization. Based on the edge map, significant line segments are detected using methods such as global Hough transform and local sensing combination to obtain an initial set of line segments. Such as using the LSD algorithm, but not limited to this.

[0069] Step S104-22: Fit the dominant line to the endpoints of the line segments in the initial line segment set to obtain the dominant structure; in this embodiment, this can be achieved by... For all line segment endpoints, the RANSAC algorithm is used to fit the direction of the dominant line. Specifically, this can be done by randomly sampling two points to determine candidate lines, and statistically determining lines whose distance is less than a threshold. The interior points; the line model with the most interior points is iteratively retained as the dominant structure of the dominant line in that direction.

[0070] Step S104-23: Construct a planar straight line diagram based on the dominant structure; in this embodiment, the planar straight line diagram PSLG can be constructed by calculating the intersection points of the dominant straight lines.

[0071] Step S104-24: Use the polygons extracted from the planar straight-line graph as the class boundary set; in this embodiment, the polygons can be extracted by traversing the interface areas of the planar straight-line graph PSLG and extracting areas with areas greater than or equal to the area threshold. Closed polygonal rings as class boundary sets Among them, the area threshold It can be defined as: in, For category The average number of point clouds.

[0072] After extracting the intra-class boundaries, it is necessary to verify whether the spatial relationships between the boundaries of objects of different categories satisfy the geometric prior conditions. Therefore, inter-class conflict detection is required. Let the intra-class boundary set (also known as the two-dimensional boundary set) be defined. , Each For an independent polygon, the corresponding 3D point cloud subset is denoted as . , Ground height, The height tolerance threshold is defined, and the specific implementation process includes: Step S104+1: Based on the intra-class boundary set, perform conflict detection on the boundaries between different categories; Step S104+2: Determine whether to correct the boundary based on the detection results; Step S104+3: Determine the modified boundary as the boundary set within the target class.

[0073] The specific implementation process of step S104+1 includes: Step S104+11: Perform penetration conflict detection and / or suspended conflict detection on the boundaries between the different categories to determine the conflict types between the different categories. In this embodiment, the conflict types may include: penetration conflict and suspended conflict. Penetration conflict can refer to the existence of geometric intersection or interpenetration relationships at the boundaries. For example, in a 3D model, two object objects that should be separate (such as a pillar and a pusher) pass through each other, resulting in an intersection relationship in the geometric structure. Suspended conflict can refer to the existence of suspended or unsupported structures at the boundaries. For example, an object object (such as a pipe or shelf) is not correctly connected to the expected supporting structure (such as a wall or floor) and is isolated and suspended. The specific implementation process of this step may include: Step S104+111: Construct a buffer for one of any two different categories of polygons; for example: for any two different categories of polygons and for Construct a buffer: in, Minkowski and, radius The closed disk, The minimum allowed distance threshold between objects of the same class is used. If this threshold is not met, the object is considered a penetration conflict.

[0074] Step S104+112: Determine whether there is an intersection between the buffer corresponding to the polygon of the stated category and the polygon of another category; that is: Check if the condition is met. If so, proceed to step S104+113.

[0075] Step S104+113: If yes, then determine that there is a conflict between the two different types of polygons, and the conflict type is a penetration conflict type; And / or, Step S104+114: Determine whether the height of the lowest point in the semantic point cloud subset corresponding to any category of polygon in the intra-class boundary set is greater than the sum of the ground height and the height tolerance threshold; that is: any polygon Determine the point cloud subset corresponding to the polygon. Does it meet the requirements? If so, then proceed to step S104+115.

[0076] Step S104+115: If yes, then determine that the polygons corresponding to the semantic point cloud subset have a conflict, and the conflict type is a dangling conflict type, i.e. This is a suspended conflict type.

[0077] Step S104+12: Perform conflict correction according to the conflict type; the specific implementation process of this step includes: Step S104+121: When the conflict type is a penetration conflict type, set one of the penetration conflict pairs as a movable point and the other as a fixed point, and calculate the signed distance from the movable point to the buffer; for example: for the detected penetration conflict pairs ,set up For movable facilities (such as shelves, display cases), For immovable structures (such as walls and load-bearing columns), calculate... (Polygon) to The signed distance of the (buffer) is given by the following formula: in, Polygon On the edge, remember for The outward normal vector at the nearest point; Step S104+122: Calculate the minimum translation correction based on the signed distance; following the example and formula 17 above, the minimum translation correction is: Step S104+123: Correct the boundary of the polygon according to the minimum translation correction amount; the corrected polygon is: .

[0078] And / or, Step S104+124: When the conflict type is a dangling conflict type, delete the polygons with dangling conflicts and the elements related to the polygons from the class boundary set. That is: using the above dangling detection results, delete the detected dangling conflict polygons. Delete the corresponding polygon from the polygon set. And its associated primitives, for example: when the detected object is a polygon of a shelf, and the legs of the shelf polygon do not touch the ground, thus causing a hanging conflict, then when deleting, all polygons representing the shelf (i.e., associated primitives) will be deleted.

[0079] Step S104-3: The process of generating the vector graphic based on the intra-class boundaries in the intra-class boundary set can be to generate the vector graphic with semantic information based on the target intra-class boundary set, for example: based on the target intra-class boundary set after the boundary correction described above, i.e., the polygon set. ,in, For set Retained and modified polygon units. Polygon set. This refers to vector graphics, which are two-dimensional structured representations, such as CAD images.

[0080] The above describes an embodiment of a vector graphic generation method provided in this application. This method, on the one hand, constructs an image dataset with semantic information. During the 3D reconstruction process based on this image dataset, it performs 3D reconstruction in a semantically guided incremental manner, resulting in 3D point cloud data with semantic information. Furthermore, the aligned point cloud data obtained by fitting the ground also contains semantic information. Semantic information is also embedded during the construction of the density map, ultimately generating a semantically aware vector graphic. This process improves the completeness and geometric accuracy of key element reconstruction in vector graphic generation application scenarios, avoiding the problem of focusing only on building structures while ignoring key elements such as shelves and pushers. It also addresses the shortcomings of easily missing sparse and small structures and sensitivity to point cloud density.

[0081] On the other hand, by introducing a semantic segmentation model, any specified task object can be accurately segmented without retraining, significantly improving semantic generalization ability, and achieving fully automatic output of vector graphics without relying on manual annotation.

[0082] On the other hand, by introducing a category-adaptive uncertainty-weighted joint optimization strategy in the semantic-aware 3D reconstruction process, key elements such as shelves and display cases are dynamically assigned higher reconstruction weights during the global joint optimization process. This enables key elements to obtain high-confidence 3D localization even under low-texture and weak observation conditions, thus improving the problem of reconstruction failure or deformation in sparse regions by existing uniform weighting methods.

[0083] Finally, in the process of generating the semantic density map, spatial relationship correction of different category boundaries is implemented, which can avoid problems such as high-density wall structures submerging sparse shelf outlines. It can also ensure that the output CAD drawing meets geometric prior constraints such as objects not penetrating each other and service elements or object facilities contacting the ground through penetration conflict detection and minimum translation correction, thereby improving the accuracy and usability of the generated CAD drawing.

[0084] The above is a detailed description of an embodiment of a vector graphic generation method provided in this application. Corresponding to the foregoing embodiment of a vector graphic generation method, this application also discloses an embodiment of a vector graphic generation apparatus. Please refer to [link to relevant documentation]. Figure 10 Since the device embodiments are basically similar to the method embodiments, the description is relatively simple, and relevant parts can be referred to in the description of the method embodiments. The device embodiments described below are merely illustrative.

[0085] like Figure 10 As shown, the device may include: Construction unit 1001 is used to construct an image dataset with semantic information based on the acquired target video data; The reconstruction unit 1002 is used to perform three-dimensional reconstruction based on the images and semantic information in the image dataset to obtain three-dimensional point cloud data with semantic information. Fitting unit 1003 is used to fit the ground based on the ground point cloud data extracted from the three-dimensional point cloud data to obtain target semantic point cloud data. The generation unit 1004 is used to generate a semantically aware vector image corresponding to the target video data based on the constructed semantic density map corresponding to the semantic category point cloud in the target semantic point cloud data.

[0086] In this embodiment, the construction unit 1001 includes: a collection subunit, an extraction subunit, a segmentation subunit, and a semantic segmentation subunit; The acquisition subunit is used to acquire panoramic video data of the target scene as the target video data; The extraction subunit is used to extract keyframe images from the target video data to obtain a keyframe image set; The segmentation unit is used to segment the keyframe images in the keyframe image set to obtain a pinhole view set; The semantic segmentation subunit is used to perform semantic segmentation on the pinhole views in the pinhole view set to obtain an image dataset with semantic information.

[0087] The molecular slicing unit includes: a definition subunit, a transformation subunit, and a determination subunit; The defined subunit is used to define the segmentation operation according to the virtual camera configuration parameter set; The transformation subunit is used to convert the keyframe images in the keyframe image set from equidistant cylindrical projection into multiple forward pinhole views according to the segmentation operation. The determining subunit is used to determine the plurality of forward pinhole views as the pinhole view set.

[0088] The semantic segmentation subunit includes: a pre-defined subunit and a construction subunit; The predefined subunit is used to predefine a set of image semantic categories; The construction subunit is used to construct the image dataset with semantic information based on the pinhole views in the pinhole view set and the image semantic category set.

[0089] Reconstruction unit 1002 includes: defining sub-units, extracting sub-units, determining sub-units, generating sub-units, and reconstructing sub-units; The definition subunit is used to define the input data of the semantic segmentation model based on the images in the image dataset and the semantic information corresponding to the images; The extraction subunit is used to input the image and the semantic information as data pairs into the geometric feature extraction submodule and the semantic feature extraction submodule of the 3D reconstruction network model, respectively extracting geometric features and semantic features corresponding to the spatial location of the geometric features; The determining subunit is used to determine the geometric features and the semantic features as a multimodal feature pair; The generating subunit is used to perform semantically gated feature matching on the multimodal feature pairs to generate a set of matching feature pairs; The reconstruction subunit is used to perform three-dimensional reconstruction on the feature pairs in the matching feature pair set using a three-dimensional reconstruction method to obtain the three-dimensional point cloud data with semantic information.

[0090] The reconstruction subunit includes: a first acquisition subunit and a second acquisition subunit; The first obtaining subunit is used to perform a three-dimensional reconstruction method based on the set of matching feature pairs to obtain an initial sparse point cloud and camera pose. The second obtaining subunit is used to perform semantic-aware joint optimization on the initial sparse point cloud and the camera pose based on the initial sparse point cloud, the camera pose, and the defined weights, so as to obtain the three-dimensional point cloud data with semantic information.

[0091] Fitting unit 1003 may include: obtaining sub-unit, extracting sub-unit, first generating sub-unit, and second generating sub-unit; The obtaining subunit is used to downsample the three-dimensional point cloud data to obtain downsampled three-dimensional point cloud data. The extraction subunit is used to extract ground point cloud datasets from the downsampled 3D point cloud data using semantic information. The first generation subunit is used to select a plane that meets the requirements from the ground point cloud dataset for ground fitting and to generate a fitted ground equation; The second generation subunit is used to perform point cloud correction on the three-dimensional point cloud data according to the fitted ground equation, and generate the target semantic point cloud data.

[0092] The first generation subunit includes: a first determination subunit, a statistics subunit, a judgment subunit, a second determination subunit, an iteration subunit, and a generation subunit; The first determining subunit is used to determine a plane based on non-collinear points randomly selected from the ground point cloud dataset; The statistical subunit is used to count the number of interior points in the ground point cloud dataset whose distance from the plane is less than or equal to a preset threshold. The judgment subunit is used to determine whether the number of interior points is greater than or equal to a preset minimum interior point number threshold. The second determining subunit is used to determine the plane as a valid candidate plane model when the judgment result of the judgment subunit is yes; The iterative subunit is used to repeat the above steps according to the number of iterations. If the number of interior points of the current valid candidate plane model is greater than or equal to the historical maximum number of interior points, then the valid candidate plane model is updated. The generating subunit is used to generate the fitted ground equation based on the updated valid candidate plane model.

[0093] The second generation subunit includes: a first obtaining subunit, a second obtaining subunit, and a generation subunit; The first obtaining subunit is used to obtain the rotation axis and rotation angle based on the fitted ground equation normal vector and the reference plane normal vector; The second obtaining subunit is used to obtain a rotation matrix based on the rotation axis and the rotation angle; The generation subunit is used to rotate and / or translate the fitted ground equation and the downsampled 3D point cloud data according to the rotation matrix to generate the target semantically aligned point cloud data.

[0094] The generation unit 1004 includes: a construction subunit, an extraction subunit, and a generation subunit; The construction subunit is used to construct a semantic density map corresponding to the semantic category point cloud based on the target semantic alignment point cloud data. The extraction subunit is used to extract intra-class boundaries based on the semantic density map to obtain an intra-class boundary set; The generation subunit is used to generate the vector diagram based on the intra-class boundaries in the intra-class boundary set.

[0095] The construction subunit includes: a projection subunit and an acquisition subunit; The projection subunit is used to project the semantic point cloud in the target semantic alignment point cloud data onto the plane of the fitted ground to generate a density map according to the semantic category of the semantic point cloud data. The obtaining subunit is used to perform density map post-processing on the density map to obtain the semantic density map.

[0096] The extraction subunit includes: a detection subunit, a fitting subunit, a construction subunit, and a determination subunit; The detection subunit is used to perform edge enhancement and line segment detection on the semantic density map to obtain an initial set of line segments. The fitting subunit is used to perform dominant line fitting on the endpoints of the line segments in the initial line segment set to obtain the dominant structure. The construction subunit is used to construct a planar straight line diagram based on the dominant structure; The determining subunit is used to take the polygons extracted from the planar straight line graph as the intraclass boundary set.

[0097] It also includes a detection subunit and a correction subunit. The detection subunit is used to perform conflict detection on the boundaries between different categories based on the intra-class boundary set, and determine whether to correct the boundary. The correction subunit is used to perform correction if the detection result of the detection subunit is yes, and determine the corrected boundary as the target intra-class boundary set.

[0098] The detection subunit includes: a judgment subunit, used to perform penetration conflict detection and / or overhang conflict detection on the boundaries between the different categories, and to determine whether a conflict exists and the type of conflict between the different categories; the correction subunit is specifically used to correct the boundary according to the conflict type when the judgment result of the judgment subunit is yes.

[0099] The determination subunit includes: a construction subunit, a first determination subunit, and a second determination subunit; and / or, a third determination subunit and a fourth determination subunit; The construction subunit is used to construct a buffer for one of any two different types of polygons; The first determining subunit is used to determine whether there is an intersection between the buffer corresponding to the polygon of the first category and the polygon of another category; The second determining subunit is used to determine, when the determination result of the first determining subunit is yes, that there is a conflict between the two polygons of different categories, and the conflict type is a penetrating conflict type; And / or, The third determining subunit is used to determine whether the height of the lowest point in the semantic point cloud subset corresponding to any type of polygon in the intra-class boundary set is greater than the sum of the ground height and the height tolerance threshold. The fourth determining subunit is used to determine that the polygon corresponding to the semantic point cloud subset has a conflict when the determination result of the third determining subunit is yes, and the conflict type is the dangling conflict type.

[0100] The correction subunit includes: a first calculation subunit, a second calculation subunit, a correction subunit; and / or, a deletion subunit; The first calculation subunit is used to, when the conflict type is a penetrating conflict type, set one of the penetrating conflict pairs as a movable point and the other as a fixed point, and calculate the signed distance from the movable point to the buffer. The second calculation subunit is used to calculate the minimum translation correction based on the signed distance; The correction subunit is used to correct the boundary of the polygon according to the minimum translation correction amount.

[0101] And / or, The deletion subunit is used to delete polygons with dangling conflicts and related elements from the class boundary set when the conflict type is a dangling conflict type.

[0102] The above is a description of an embodiment of a vector graphic generation apparatus provided in this application. For details of the apparatus embodiment, please refer to the relevant content of the above method embodiment.

[0103] Based on the above, this application also provides a computer storage medium, including a computer program, which, when run on an electronic device, causes the electronic device to perform the steps in the vector graphic generation method described above.

[0104] Based on the above, this application also provides an electronic device, such as... Figure 11 As shown, it includes: processor 1101; The memory 1102 is used to store a program for processing data generated by the electronic device. When the program is read and executed by the processor 1101, it performs the steps in the vector graphic generation method described above.

[0105] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0106] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0107] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0108] 1. Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0109] 2. Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] Although this application discloses preferred embodiments as described above, it is not intended to limit this application. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of this application. Therefore, the scope of protection of this application should be determined by the scope defined in the claims of this application.

Claims

1. A method for generating vector graphics, characterized in that, include: Based on the collected target video data, an image dataset with semantic information is constructed. Three-dimensional reconstruction is performed based on the images and semantic information in the image dataset to obtain three-dimensional point cloud data with semantic information. The ground is fitted with ground point cloud data extracted from the three-dimensional point cloud data to obtain target semantically aligned point cloud data; Based on the semantic density map corresponding to the semantic category point cloud in the constructed point cloud data aligned with the target semantics, a semantically aware vector image corresponding to the target video data is generated.

2. The method according to claim 1, characterized in that, The construction of an image dataset with semantic information based on the acquired target video data includes: Collect panoramic video data of the target scene as the target video data; Extract keyframe images from the target video data to obtain a keyframe image set; The keyframe images in the keyframe image set are segmented to obtain a pinhole view set; Semantic segmentation is performed on the pinhole views in the pinhole view set to obtain an image dataset with semantic information.

3. The method according to claim 2, characterized in that, The step of segmenting the keyframe images in the keyframe image set to obtain a pinhole view set includes: The segmentation operation is defined based on the virtual camera configuration parameter set; According to the segmentation operation, the keyframe images in the keyframe image set are converted from equidistant cylindrical projection into multiple forward pinhole views; The plurality of forward pinhole views are defined as the pinhole view set.

4. The method according to claim 2, characterized in that, The step of semantically segmenting the pinholes in the pinhole view set to obtain an image dataset with semantic information includes: A predefined set of image semantic categories; The image dataset with semantic information is constructed based on the pinhole views in the pinhole view set and the image semantic category set.

5. The method according to claim 1, characterized in that, The step of performing 3D reconstruction based on images and semantic information in the image dataset to obtain 3D point cloud data with semantic information includes: The input data for the semantic segmentation model is defined based on the images in the image dataset and the semantic information corresponding to the images. The image and the semantic information are used as a data pair and input into the geometric feature extraction submodule and the semantic feature extraction submodule of the 3D reconstruction network model to extract geometric features and semantic features corresponding to the spatial location of the geometric features, respectively. The geometric features and the semantic features are identified as a multimodal feature pair; Semantic-gated feature matching is performed on the multimodal feature pairs to generate a set of matching feature pairs; The feature pairs in the matching feature pair set are reconstructed using a 3D reconstruction method to obtain the 3D point cloud data with semantic information.

6. The method according to claim 5, characterized in that, The step of performing 3D reconstruction on the feature pairs in the matching feature pair set to obtain the 3D point cloud data with semantic information includes: Based on the set of matching feature pairs, a 3D reconstruction method is performed to obtain an initial sparse point cloud and camera pose. Based on the initial sparse point cloud, the camera pose, and the defined weights, semantic-aware joint optimization is performed on the initial sparse point cloud and the camera pose to obtain the 3D point cloud data with semantic information.

7. The method according to claim 1, characterized in that, The step of fitting the ground with ground point cloud data extracted from the 3D point cloud data to obtain target semantic point cloud data includes: The three-dimensional point cloud data is downsampled to obtain downsampled three-dimensional point cloud data; Ground point cloud datasets are extracted from the downsampled 3D point cloud data using semantic information. Select planes that meet the requirements from the ground point cloud dataset to perform ground fitting and generate the fitted ground equation; The three-dimensional point cloud data is corrected according to the fitted ground equation to generate the target semantic point cloud data.

8. The method according to claim 7, characterized in that, The step of selecting a plane that meets the requirements from the ground point cloud dataset for ground fitting and generating a fitted ground equation includes: The plane is determined based on non-collinear points randomly selected from the ground point cloud dataset; Count the number of interior points in the ground point cloud dataset whose distance from the plane is less than or equal to a preset threshold; Determine whether the number of interior points is greater than or equal to a preset minimum interior point threshold; If so, the plane is determined as a valid candidate plane model; During the process of repeating the above steps according to the number of iterations, if the number of interior points of the current valid candidate plane model is greater than or equal to the historical maximum number of interior points, then the valid candidate plane model is updated. The fitted ground equation is generated based on the updated valid candidate plane model.

9. The method according to claim 7 or 8, characterized in that, The step of performing point cloud correction on the 3D point cloud data according to the fitted ground equation to generate the target semantically aligned point cloud data includes: Based on the fitted ground equation normal vector and the reference plane normal vector, the rotation axis and rotation angle are obtained; Based on the rotation axis and the rotation angle, obtain the rotation matrix; The fitted ground equation and the downsampled 3D point cloud data are rotated and / or translated according to the rotation matrix to generate the target semantically aligned point cloud data.

10. The method according to claim 1, characterized in that, The step of generating a semantically aware vector map corresponding to the target video data based on the semantic density map corresponding to the semantic category point cloud in the constructed point cloud data aligned with the target semantics includes: Based on the target semantic alignment point cloud data, construct a semantic density map corresponding to the semantic category point cloud; Based on the semantic density map, extract the intra-class boundaries to obtain the intra-class boundary set; The vector diagram is generated based on the intra-class boundaries in the intra-class boundary set.

11. The method according to claim 10, characterized in that, The step of constructing a semantic density map corresponding to the semantic category point cloud based on the target semantic aligned point cloud data includes: According to the semantic category of the semantic point cloud in the target semantic alignment point cloud data, a density map is generated by projecting it onto the plane of the fitted ground. The density map is post-processed to obtain the semantic density map.

12. The method according to claim 10, characterized in that, The step of extracting intra-class boundaries to obtain an intra-class boundary set based on the semantic density map includes: Edge enhancement and line segment detection are performed on the semantic density map to obtain an initial set of line segments; The dominant structure is obtained by fitting dominant lines to the endpoints of the line segments in the initial set of line segments. Construct a planar line diagram based on the dominant structure; The polygons extracted from the planar straight line graph are used as the intra-class boundary set.

13. The method according to claim 10 or 12, characterized in that, Also includes: Based on the intra-class boundary set, conflict detection is performed on the boundaries between different categories to determine whether the boundaries need to be corrected. If so, the correction is performed, and the corrected boundary is determined as the boundary set within the target class.

14. The method according to claim 13, characterized in that, The step of performing conflict detection on the boundaries between different categories based on the intra-class boundary set, and determining whether to correct the boundaries, includes: Penetration conflict detection and / or overhang conflict detection are performed on the boundaries between the different categories to determine whether a conflict exists between the different categories and the type of conflict. If so, the boundary is corrected according to the conflict type.

15. The method according to claim 14, characterized in that, The step of performing penetration conflict detection and / or overhang conflict detection between the different categories to determine whether a conflict exists and the type of conflict between the different categories includes: Construct a buffer for one of any two different classes of polygons; Determine whether there is an intersection between the buffer corresponding to the polygon of the stated category and the polygon of another category; If so, then it is determined that there is a conflict between the two different types of polygons, and the conflict type is a penetration conflict type; And / or, Determine whether the height of the lowest point in the semantic point cloud subset corresponding to any category of polygon in the intra-class boundary set is greater than the sum of the ground height and the height tolerance threshold. If so, it is determined that the polygons corresponding to the semantic point cloud subset have a conflict, and the conflict type is a dangling conflict type.

16. The method according to claim 14, characterized in that, The step of correcting the boundary according to the conflict type includes: When the conflict type is a penetration conflict type, one of the penetration conflict pairs is set as a movable point and the other is set as a fixed point, and the signed distance from the movable point to the buffer is calculated; Calculate the minimum translation correction based on the signed distance; The boundary of the polygon is corrected according to the minimum translation correction amount; And / or, When the conflict type is a dangling conflict type, the polygons with dangling conflicts and the elements associated with the polygons are deleted from the class boundary set.

17. A vector graphic generation apparatus, characterized in that, include: The building unit is used to construct an image dataset with semantic information based on the acquired target video data; The reconstruction unit is used to perform three-dimensional reconstruction based on the images and semantic information in the image dataset to obtain three-dimensional point cloud data with semantic information. The fitting unit is used to fit the ground based on the ground point cloud data extracted from the three-dimensional point cloud data to obtain the target semantic point cloud data. The generation unit is used to generate a semantically aware vector image corresponding to the target video data based on the constructed semantic density map corresponding to the semantic category point cloud in the target semantic point cloud data.

18. A computer storage medium, characterized in that, The method includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the method as described in any one of claims 1 to 16.

19. An electronic device, characterized in that, include: processor; A memory for storing a program for processing data generated by an electronic device, wherein when the program is read and executed by the processor, it performs the steps of the method as described in any one of claims 1 to 16.

Images