3D Vector Mesh Generation Method and Device

Abstract

The present application discloses a 3D vector grid generation method and device. The method includes: obtaining multiple first images for a current scene, performing point cloud reconstruction on the multiple first images to obtain the point cloud of the current scene and the poses of the first images; performing semantic segmentation on each of the multiple first images to obtain the 2D semantic information of the first image; generating a 3D semantic model of the current scene according to the point cloud of the current scene, the poses of the first images, and the 2D semantic information of each of the multiple first images; performing feature extraction and classification of a vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene; and obtaining the 3D vector grid of the current scene according to the category and parameters of the vector model in the current scene. The solution of the embodiment can construct 3D vector grids for various scenes, greatly improving the applicable range and commercial value of vector modeling.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality (AR), and in particular, to a method and device for generating a 3D vector mesh. Background Art

[0002] With the development of AR technology, a brand-new digital reality experience of virtual-real fusion has entered the lives of ordinary users. The AR ecosystem has gradually become rich, and new AR applications have emerged in an endless stream. Among them, the interaction related to the real-world scene is inseparable from the construction of the three-dimensional information of the scene.

[0003] Such as Figure 1 As shown, as a concise representation of the three-dimensional information of the scene, the 3D vector mesh plays an important role in virtual object placement, scene interaction collision, virtual-real occlusion, etc. Its acquisition method is mainly to measure the 3D information of the scene by a professional surveying and mapping team, and then manually processed by professionals to generate it, with low efficiency and high cost.

[0004] Currently, for the method of automatically generating a vector mesh, it mainly relies on 3D plane fitting, and can only generate the vector mesh of the outer contour of planar buildings, which is not applicable to curved buildings and indoor scenes. By performing 2D semantic segmentation on an image sequence through a convolutional neural network (CNN) network, etc., the shape annotation of indoor and outdoor images can be realized, and then through optimization methods such as a Markov random field (MRF), etc., an indoor and outdoor 3D mesh with semantic information can be generated. However, the shape of the 3D semantic mesh has not changed compared with the original mesh, with a large amount of data and complex topology. Therefore, it is mainly used for navigation, scene understanding, target recognition, etc., and the scene structure information it contains has not been widely developed for applications in modeling. Summary of the Invention

[0005] Embodiments of the present application provide a method and device for generating a 3D vector mesh. By adopting the solution of the embodiments, 3D vector meshes of various scenes, such as indoor, outdoor, planar, and curved surfaces, etc., can be constructed, greatly improving the applicable range and commercial value of vector modeling.

[0006] In a first aspect, embodiments of the present application provide a method for generating a 3D vector mesh, including:

[0007] Obtain multiple first images for the current scene, where the multiple first images are consecutive frames; perform point cloud reconstruction on the multiple first images to obtain the point cloud of the current scene and the pose of each first image in the multiple first images; perform semantic segmentation on each first image in the multiple first images to obtain the 2D semantic information of the first image; generate a 3D semantic model of the current scene according to the point cloud of the current scene, the pose of each first image in the multiple first images, and the 2D semantic information of each first image in the multiple first images, where the 3D semantic model of the current scene is a 3D point cloud carrying category information; perform feature extraction and classification of the vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene; obtain the 3D vector grid of the current scene according to the category and parameters of the vector model in the current scene.

[0008] Among them, the 3D semantic model of the current scene being a 3D point cloud carrying category information specifically means that the 3D semantic model of the current scene includes a 3D point cloud and the category of each 3D point in the 3D point cloud. For example, if there is a table in the current scene, in the 3D semantic model of the current scene, the category of each 3D point corresponding to the table in the current scene is a table.

[0009] Obtaining the 3D vector grid of the current scene according to the category and parameters of the vector model in the current scene specifically includes obtaining the vector model of the current scene from the vector model library according to the vector model and parameters of the current scene, and the 3D vector grid of the current scene includes the vector model of the current scene.

[0010] Optionally, the number of vector models in the current scene can be one or more; the 3D vector grid of the current scene includes one or more vector models in the current scene.

[0011] By combining shape semantic segmentation, the problem that vector modeling can only reconstruct planar information is solved, and at the same time, the problem that existing vector modeling cannot adapt to complex indoor scenes is solved; by performing feature extraction and classification of the vector model based on the 3D semantic model of the current scene to obtain the category and parameters of the vector model, and obtaining the vector model of the current scene from the vector model library according to the category and parameters of the vector model, it is possible to reconstruct a variety of vector models relying on the 3D semantic model, thereby making it possible to obtain a comprehensive and accurate 3D vector grid, truly opening up the path between the 3D semantic model and the 3D vector grid, enabling the system to adapt to various scenes, such as indoor, outdoor, planar, and curved surfaces, etc., greatly improving the applicable range and commercial value of vector modeling.

[0012] In a feasible embodiment, performing feature extraction and classification of the vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene includes:

[0013] Extract features from the 3D semantic model of the current scene to obtain the 3D feature map of the current scene; perform feature decoding on the 3D feature map of the current scene to obtain the feature vectors of the vector model in the 3D semantic model of the current scene; classify the vector model according to the feature vectors of the vector model to obtain the category of the vector model; obtain the parameters of the vector model according to the 3D feature map of the current scene.

[0014] By obtaining the type and parameters of the vector model in the current scene based on the 3D semantic model of the current scene, the 3D vector modeling is not limited to several structural categories of features, and can adapt to various scenes such as indoor, outdoor, plane and curved surfaces, greatly improving the application range and commercial value of vector modeling.

[0015] In a feasible embodiment, performing feature extraction and classification of the vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene, including:

[0016] Call a multi-task neural network, input the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification, and obtain the category and parameters of the vector model in the current scene.

[0017] In a feasible embodiment, the multi-task neural network includes a convolutional layer, a first fully-connected layer, a second fully-connected layer and a classifier. Among them, the output data of the convolutional layer is the input data of the first fully-connected layer and the input data of the second fully-connected layer, and the output data of the first fully-connected layer is the input data of the classifier. Input the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification, and obtain the category and parameters of the vector model in the current scene, including:

[0018] Input the 3D semantic model of the current scene into the convolutional layer for feature extraction to obtain the 3D feature map of the current scene; input the 3D feature map of the current scene into the first fully-connected layer for feature decoding to obtain the feature vectors of the vector model in the 3D semantic model of the current scene; input the feature vectors of the vector model in the 3D semantic model of the current scene into the classifier for classification to obtain the category of the vector model in the 3D semantic model of the current scene; input the 3D feature map of the current scene into the second fully-connected layer for processing to obtain the parameters of the vector model in the 3D semantic model of the current scene.

[0019] By introducing a multi-task neural network and inputting the 3D semantic model, the category and parameters of the vector model are obtained, so as to obtain a more refined vector result and expand the application range.

[0020] In a feasible embodiment, the convolutional layer includes a first convolutional layer and a second convolutional layer, and the structure and parameters of the first convolutional layer are the same as those of the second convolutional layer;

[0021] The output data of the first convolutional layer serves as the input data for the first fully connected network; the output data of the second convolutional layer serves as the output data for the second fully connected layer; the 3D semantic model of the current scene is input into the first convolutional layer for feature extraction to obtain the 3D feature map of the current scene; the 3D semantic model of the current scene is input into the second convolutional layer for feature extraction to obtain the 3D feature map of the current scene.

[0022] Among them, since the first convolutional layer and the second convolutional layer share weights during the feature extraction stage, the 3D feature map of the first convolutional layer is the same as the 3D feature map output by the second convolutional layer.

[0023] In a feasible embodiment, generating the 3D semantic model of the current scene according to the point cloud of the current scene, the pose of each first image among multiple first images, and the 2D semantic information of each first image among multiple first images includes:

[0024] Projecting the target point cloud onto each of the multiple first images according to the pose of each first image among the multiple first images to obtain the category corresponding to each 3D point in the target point cloud on the first image; wherein, the target point cloud is the point cloud of the current scene; using the 3D points in the target point cloud as nodes and the connection lines between the 3D points and the corresponding categories of the 3D points on the first image as edges to construct a graph model based on the Markov random field (MRF); performing MRF optimization and solution on the graph model based on the MRF according to the energy function to obtain the 3D semantic model of the current scene, and each 3D point in this 3D semantic model has a unique category.

[0025] When projecting the target point cloud onto the multiple first images, since a 3D point in the target point cloud may be projected onto multiple first images, a 3D point in the target point cloud may correspond to multiple categories. Therefore, a graph model based on the MRF is constructed according to the above method, and then the graph model is optimized and solved according to the energy function, so as to obtain a 3D semantic model in which the 3D points have unique categories.

[0026] In a second aspect, an embodiment of the present application provides a server, including:

[0027] An acquisition unit, configured to acquire multiple first images for the current scene, and the multiple first images are consecutive frames;

[0028] A processing unit, configured to perform point cloud reconstruction on the multiple first images to obtain the point cloud of the current scene and the pose of each first image among the multiple first images; perform semantic segmentation on each first image among the multiple first images to obtain the 2D semantic information of the first image;

[0029] A generation unit for generating a 3D semantic model of the current scene based on the point cloud of the current scene, the poses of each first image among multiple first images, and the 2D semantic information of each first image among multiple first images. The 3D semantic model of the current scene is a 3D point cloud carrying category information.

[0030] A feature extraction and classification unit for performing feature extraction and classification of a vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene.

[0031] An acquisition unit is also used to obtain the 3D vector grid of the current scene according to the category and parameters of the vector model in the current scene.

[0032] In a feasible embodiment, the feature extraction and classification unit is specifically used for:

[0033] Performing feature extraction on the 3D semantic model of the current scene to obtain a 3D feature map of the current scene; performing feature decoding on the 3D feature map of the current scene to obtain the feature vector of the vector model in the 3D semantic model of the current scene; classifying the vector model according to the feature vector of the vector model to obtain the category of the vector model; obtaining the parameters of the vector model according to the 3D feature map of the current scene.

[0034] In a feasible embodiment, the feature extraction and classification unit is specifically used for:

[0035] Invoking a multi-task neural network, inputting the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification to obtain the category and parameters of the vector model in the current scene.

[0036] In a feasible embodiment, the multi-task neural network includes a convolutional layer, a first fully connected layer, a second fully connected layer, and a classifier. Among them, the output data of the convolutional layer is the input data of the first fully connected layer and the input data of the second fully connected layer, and the output data of the first fully connected layer is the input data of the classifier. In terms of inputting the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification to obtain the category and parameters of the vector model in the current scene, the feature extraction and classification unit is specifically used for:

[0037] Inputting the 3D semantic model of the current scene into the convolutional layer for feature extraction to obtain a 3D feature map of the current scene; inputting the 3D feature map of the current scene into the first fully connected layer for feature decoding to obtain the feature vector of the vector model in the 3D semantic model of the current scene; inputting the feature vector of the vector model in the 3D semantic model of the current scene into the classifier for classification to obtain the category of the vector model in the 3D semantic model of the current scene; inputting the 3D feature map of the current scene into the second fully connected layer for processing to obtain the parameters of the vector model in the 3D semantic model of the current scene.

[0038] In a feasible embodiment, the convolutional layer includes a first convolutional layer and a second convolutional layer, and the structure and parameters of the first convolutional layer are the same as those of the second convolutional layer; the output data of the first convolutional layer is the input data of the first fully-connected network; the output data of the second convolutional layer is the output data of the second fully-connected layer; in terms of inputting the 3D semantic model of the current scene into the convolutional layer for feature extraction to obtain the 3D feature map of the current scene, the feature extraction and classification unit is specifically used for:

[0039] Input the 3D semantic model of the current scene into the first convolutional layer for feature extraction to obtain the 3D feature map of the current scene; input the 3D semantic model of the current scene into the second convolutional layer for feature extraction to obtain the 3D feature map of the current scene.

[0040] In a feasible embodiment, the generating unit is specifically used for:

[0041] Project the target point cloud onto each of the multiple first images according to the pose of each first image in the multiple first images to obtain the category corresponding to each 3D point in the target point cloud on the first image; wherein, the target point cloud is the point cloud of the current scene; use the 3D points in the target point cloud as nodes, and use the connection lines between the 3D points and the categories corresponding to the 3D points on the first image as edges to construct a graph model based on the Markov random field (MRF); perform MRF optimization solution on the graph model based on the MRF according to the energy function to obtain the 3D semantic model of the current scene, and each 3D point in the 3D semantic model has a unique category.

[0042] In a third aspect, an embodiment of the present application provides a server, which is characterized by including:

[0043] A memory for storing instructions; and

[0044] A processor coupled to the memory;

[0045] Wherein, when the processor executes the instructions, it executes part or all of the method described in the first aspect.

[0046] In a fourth aspect, an embodiment of the present application provides a chip system, which is characterized in that the chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; the interface circuits are used to receive signals from the memory of the electronic device and send the signals to the processors, and the signals include computer instructions stored in the memory; when the processors execute the computer instructions, the electronic device executes part or all of the method described in the first aspect.

[0047] Fifth aspect, an embodiment of the present application provides a computer storage medium, characterized in that the computer storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute part or all of the method described in the first aspect.

[0048] Sixth aspect, an embodiment of the present application provides a computer program product, characterized in that it includes computer instructions, and when the computer instructions run on an electronic device, the electronic device is caused to execute part or all of the method described in the first aspect.

[0049] These aspects or other aspects of the present invention will be more clearly understood in the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0051] Figure 1 It is a schematic diagram of a 3D vector grid;

[0052] Figure 2 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0053] Figure 3 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0054] Figure 4 It is another schematic diagram of an application scenario provided by an embodiment of the present application;

[0055] Figure 5 It is a schematic flowchart of a 3D vector grid generation method provided by an embodiment of the present application;

[0056] Figure 6 It is a schematic diagram of the architecture of a multi-task neural network provided by an embodiment of the present application;

[0057] Figure 7 It is another schematic diagram of the architecture of a multi-task neural network provided by an embodiment of the present application;

[0058] Figure 8 It is a schematic diagram of the architecture of a server provided by an embodiment of the present application;

[0059] Figure 9 It is another schematic diagram of the architecture of a server provided by an embodiment of the present application. Detailed implementation manners

[0060] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0061] See Figure 2 , Figure 2 which is a schematic diagram of a system architecture provided by an embodiment of the present application. As Figure 2 shown, the system architecture includes a terminal device 201 and a server 202;

[0062] Among them, the above-mentioned terminal device 201, also known as a user equipment (UE), is a device that provides voice and / or data connectivity to users. For example, a handheld device with a wireless connection function and a camera, a vehicle-mounted device, etc. Common terminals include, for example: mobile phones, tablet computers, laptop computers, palmtop computers, mobile internet devices (MIDs) with cameras, and wearable devices, such as smart watches with cameras, smart glasses, etc.

[0063] The above-mentioned server 202 is used to receive, process, and store data.

[0064] The terminal device 201 continuously collects multiple images for the current scene, and sends the multiple images to the server 202. The server obtains the 3D vector grid of the current scene according to the multiple images according to the method of the present application; the server 202 sends the 3D vector grid of the current scene to the terminal device 201; the terminal device 201 uses the 3D vector grid of the current scene for rendering.

[0065] The terminal device 201 realizes the interaction of the scene in the virtual reality and augmented reality scenarios based on the 3D vector grid of the current scene. For example, placing virtual objects in the current scene. By loading the 3D vector grid generated by the present invention, based on the 3D vector network, virtual objects such as red lanterns, hot air balloons, and dancing fruits and vegetables can be preset or selected by the user to be placed in the scene, and then combined with the real scene for rendering and display, so as to achieve the purpose of combining virtual and real. Through the three-dimensional information of the scene provided by the 3D vector grid, it can be ensured that the placed objects seem to be real objects in the real scene, as Figure 3 shown.

[0066] For another example, in an AR application, by loading the 3D vector grid generated by the present invention, the user can interact with the scene, and then combined with the real scene for rendering and display, so as to achieve the purpose of combining virtual and real. Through the three-dimensional information of the scene provided by the 3D vector grid, it can be ensured that the virtual objects thrown by the user "hit" the real scene truly, providing a better AR experience, as Figure 4 shown, where a virtual basketball collides with a real table.

[0067] In another example, after the above terminal device 201 obtains multiple images, according to the processing method of the multiple images by the server 202, the terminal device 201 processes the multiple images to obtain a 3D vector grid of the current scene, and then performs rendering based on the 3D vector grid of the current scene.

[0068] The following introduces how the server obtains the 3D vector grid of the current scene based on the multiple images uploaded by the terminal device.

[0069] See Figure 5 , Figure 5 which is a schematic flowchart of a 3D vector grid generation method provided by an embodiment of the present application. As Figure 5 shown, the method includes:

[0070] S501. Obtain multiple first images, where the multiple first images are consecutive frames.

[0071] Among them, the multiple first images may be multiple images taken at different angles of the same scene. The image is an RGB format image or an image in other formats.

[0072] S502. Perform point cloud reconstruction on the current scene according to the multiple first images to obtain the point cloud of the current scene and the pose of each of the multiple first images.

[0073] Specifically, performing point cloud reconstruction on the current scene according to the multiple first images to obtain the point cloud of the current scene and the pose of each of the multiple first images includes:

[0074] Feature extraction is performed on each of the multiple first images to obtain multiple feature descriptors for each first image, and the multiple feature descriptors of each first image correspond one-to-one to multiple feature points in the first image; wherein, the method of performing feature extraction on each of the multiple first images may be the scale-invariant feature transform (SIFT) method; multiple image pairs are obtained from the multiple first images, and any one of the multiple image pairs includes any two first images among the multiple first images, or two first images with a time difference less than a first preset threshold during acquisition among the multiple first images; according to the feature descriptors of the images in each image pair among the multiple image pairs, the feature points of the images in the image pair are matched to obtain one or more first feature point matching pairs, and each first feature point matching pair in the one or more first feature point matching pairs includes a first feature point and a second feature point, and the first feature point and the second feature point included in the feature point pair are respectively two first images in the image pair to which the first feature point matching pair belongs, and the Euclidean distance between the feature descriptor corresponding to the first feature point and the feature descriptor corresponding to the second feature point is less than a second preset threshold; for one or more first feature point matching pairs of the image pair, through the epipolar geometry relationship of the feature point matching pair, the essential matrix, the fundamental matrix, and the homography matrix are estimated using the random sample consensus algorithm, and based on the estimated essential matrix, fundamental matrix, and homography matrix, it is determined whether the first feature point matching pair is a mismatched feature point pair; the mismatched feature point pairs in one or more feature point matching pairs of the image pair are deleted according to this method to obtain one or more second feature point matching pairs of the image pair. At this point, the second feature point matching pairs of each image pair are all accurate matching pairs;

[0075] A target image pair is obtained from the multiple image pairs, and the target image pair is an image pair that meets the preset conditions. Among them, the preset conditions are: the number of second feature point matching pairs in the image pair is greater than a first preset number, the front-back translation distance between the two first images in the target image pair is less than a first preset distance, and the triangulation angle of the target image pair is greater than a first preset angle; the pose of the first image with an earlier acquisition time in the target image pair is calculated, a first coordinate system is constructed based on the pose of the first image, and the pose of the other first image in the target image pair in the first coordinate system is calculated; and the feature point matching pairs of the target image pair are triangulated to obtain 3D points corresponding to the feature point matching pairs, and the 3D points form an initial point cloud; at this time, the two first images in the target image pair can be regarded as registered images;

[0076] Sort the images associated with the registered image according to the optimal view frame selection method to obtain a sorting result. Among them, the images associated with the registered image include the images that form the above-mentioned image pairs with the registered image; obtain the next target frame from the sorting result. The target frame is the one with a higher ranking in the sorting result and the image pair formed with the registered image satisfies the above-mentioned preset conditions; use the p3p random sample consensus algorithm to calculate the pose of the target frame, realizing the registration of the target frame. The pose of the target frame is the pose in the first coordinate system; according to the pose of the target frame and the pose of the registered image, and the feature point matching pairs between the target frame and the registered image, triangulation is performed to obtain the 3D points corresponding to the feature point matching pairs; the 3D points and the 3D points in the initial point cloud form the first point cloud; then repeat the above steps until the above-mentioned multiple first images are all registered and triangulated to obtain the target point cloud;

[0077] Perform bundle adjustment (BA) on the 3D points in the target point cloud and the poses of the above-mentioned registered images. The method used is to optimize the poses and the 3D point cloud by minimizing the reprojection error; specifically, the input is the registered images and the 3D points in the target point cloud, and the output is the optimized pose of each image in the registered images and the optimized 3D points. The point cloud formed by the optimized 3D points is a sparse point cloud. Optionally, to ensure accuracy, re-triangulation and filtering operations are performed on the optimized 3D points, and the point cloud formed by the obtained 3D points is a sparse point cloud.

[0078] In an optional embodiment, after performing the registration and triangulation of the target frame, perform bundle adjustment on the obtained 3D points and the pose of the target frame; then register and triangulate the next frame according to the above method, and then perform bundle adjustment on the obtained 3D points and the pose of the next frame until the registration, triangulation, and bundle adjustment of all the first images are completed to obtain the above-mentioned sparse point cloud and the pose of each first image.

[0079] Based on the poses and co-visibility relationships of each first image in the multiple first images, filter out the reliable neighbor frames of each first image in the multiple first images. The neighbor frames satisfy that the baseline is long enough (to ensure accuracy) and the common visible area is large enough (to ensure coverage).

[0080] Based on the above sparse point cloud, an initial value is assigned to each depth pixel in the corresponding depth map of the point cloud; each pixel point in the initial depth map is traversed, and the depth value of each pixel is iteratively optimized using neighborhood propagation and random perturbation with the normalized cross correlation (NCC) similarity score of the image patch (such as an image patch with a size of 7*7) where the pixel point is located as the criterion, and unreliable depth values are removed to obtain the original depth map; the original depth map is subjected to depth debris removal and small hole / gap filling operations to obtain an optimized depth map;

[0081] Since the above optimized depth map is not yet ideal enough, the optimized depth map needs to be further optimized, specifically including: projecting the depth map corresponding to the neighborhood frame of the current image onto the current image according to the pose of the current image to obtain a new depth map, and determining the confidence of the neighborhood frame of the current image as the confidence of the new depth map; determining whether the depth value corresponding to the neighborhood frame of the current image is similar to the depth value of the current image; if similar, adding the weighted depth of the neighborhood frame of the current image, where the weighted depth specifically refers to the product of the depth value and the confidence, and updating the positive frame counter and the accumulated positive confidence; otherwise, updating the negative counter and the negative confidence, that is, when the depth values are similar, the positive frame counter is incremented by 1, and when they are not similar, the negative frame counter is incremented by 1;

[0082] If the positive frame counter is greater than the third preset threshold, and the sum of the accumulated positive confidence is greater than the sum of the accumulated negative confidence, and the final weighted depth is greater than the minimum depth and less than the maximum depth, then update the depth and confidence of this point; otherwise, discard the depth and confidence of this point, and discarding the confidence of this point specifically means setting the confidence to 0.

[0083] According to the above method, the further optimization of the above optimized depth map is completed to obtain the target depth map.

[0084] The target depth map is converted into a three-dimensional point cloud with scale information through the internal parameters of the depth map, and the three-dimensional point cloud with scale information is converted into the scene coordinate system according to the pose of each first image in the multiple first images and fused with the previous sparse point cloud to obtain a complete point cloud that can describe the scene, and this point cloud is the target point cloud.

[0085] S503. Perform shape semantic segmentation on each of the multiple first images to obtain the 2D semantic information of each of the first images.

[0086] Specifically, feature extraction is performed on each of the multiple first images to obtain multiple first feature maps of the first image; the scales of the multiple first feature maps are different; the multiple first feature maps of the first image are feature-fused to obtain a second feature map of the first image; each pixel point in the second feature map is classified to obtain the category of each pixel point.

[0087] Among them, for feature extraction of each first image, network structures such as VGGNet, ResNet, ResNXet, DenseNet, etc. can be specifically used to extract the abstract semantic features of different scales of the first image; for feature fusion of the extracted abstract semantic features of different scales of the first image, it includes upsampling the first feature map with the lowest resolution among the multiple first feature maps of the first image to obtain a third feature map, and the resolution of the third feature map is N times the resolution of the first feature map with the lowest resolution; fusing the third feature map with the feature map having the same resolution as the third feature map among the multiple first feature maps to obtain a fourth feature map; upsampling the fourth feature map to obtain a fifth feature map, and the resolution of the fifth feature map is N times the resolution of the fourth feature map; fusing the fifth feature map with the feature map having the same resolution as the fifth feature map among the multiple first feature maps to obtain a sixth feature map; repeatedly executing the upsampling and fusion processes until a feature map with the same resolution as the maximum resolution among the multiple first feature maps is fused, and this feature map is the target feature map of the first image;

[0088] Since the upsampling multiple in the above upsampling process is high, resulting in relatively blurred semantic features and loss of details in the image, problems such as incomplete segmentation will eventually occur. Therefore, the following method can be adopted:

[0089] Use DeepLab-v3+ to perform semantic segmentation on each of the multiple first images, and introduce an atrous spatial pyramid pooling (ASPP) structure based on dilated convolution in the DeepLab-v3+ structure, thereby introducing multi-scale information, so as to obtain multi-scale feature maps of each first image; then further fuse the low-scale feature map and the high-scale feature map to obtain the target feature map of the first image, and this method improves the accuracy of segmentation. In addition, in order to improve the segmentation speed and robustness, depthwise separable convolution is introduced in ASPP and feature upsampling and fusion.

[0090] After obtaining the target feature maps of each of the multiple first images, each pixel in the target feature map of each first image is classified to obtain the category of each pixel point; the classification of the pixel points in the target feature map can be implemented using a softmax classifier; the 2D semantic information of each of the above first images includes the category of each pixel point in the target feature map of the first image.

[0091] S504. According to the point cloud of the current scene and the poses of each of the multiple first images, and the 2D semantic information of each of the multiple first images, obtain the 3D semantic model of the current scene.

[0092] Specifically, through the visible graph model of structure-from-motion (SFM), project the target point cloud onto each of the multiple first images according to each of the multiple first images, and obtain the category corresponding to each 3D point in the target point cloud on the first image; in order to realize the transfer of 2D semantics to 3D semantic point cloud, use the 3D point cloud as nodes and the connection lines between the 3D points in the point cloud and the corresponding categories of the 3D points in the first image as edges to construct a graph model based on MRF; since in the process of semantic label transfer between 2D semantics and 3D points, multiple 2D superpixels corresponding to a 3D point may have different label values, that is, a 3D point may correspond to multiple categories, by constructing an energy function and using MRF to optimize and solve, obtain the final point cloud carrying 3D semantics, and this point cloud carrying 3D semantics is the 3D semantic model of the current scene, and each 3D point in this 3D point cloud has a unique category.

[0093] Among them, the above energy function is well-known to those skilled in the art and will not be specifically described here.

[0094] S505. Perform feature extraction and classification according to the 3D semantic model of the current scene to obtain the type and parameters of the vector model in the current scene, and obtain the 3D vector grid of the current scene according to the type and parameters of the vector model in the current scene.

[0095] In a feasible embodiment, obtaining the 3D vector grid of the current scene according to the 3D semantic model of the current scene includes:

[0096] Perform feature extraction on the 3D semantic model of the current scene to obtain the 3D feature map of the current scene; perform feature decoding on the 3D feature map of the current scene to obtain the feature vector of the vector model in the 3D semantic model of the current scene; classify the vector model according to the feature vector of the vector model to obtain the category of the vector model; obtain the parameters of the vector model according to the 3D feature map of the current scene, and the 3D vector grid of the current scene includes the category and parameters of the vector model.

[0097] Among them, the vector model is a model with vector features in the 3D feature map of the current scene. The vector features can be, for example, straight lines, curves, planes, etc. The category of the vector model is the category of the object corresponding to the vector model in the current scene, such as tables, chairs, balloons, doors, trees, etc.; the parameters of the vector model are the parameters of the object corresponding to the vector model in the current scene, such as the height, position, material, etc. of the table, and the position, size, etc. of the door.

[0098] Optionally, a 3D vector grid of the current scene is obtained according to the 3D semantic model of the current scene, including:

[0099] Call a multi-task neural network, input the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification, and obtain the category and parameters of the vector model in the current scene.

[0100] As Figure 6 shown, the multi-task neural network includes a convolutional layer, two fully connected layers and a classifier. The two fully connected layers are the first fully connected layer and the second fully connected layer respectively; the output data of the convolutional layer is the input data of the two fully connected layers, and the output data of the first fully connected layer is the input data of the classifier; input the 3D semantic model of the current scene into the convolutional layer, perform feature extraction on the 3D semantic model of the current scene, and obtain the 3D feature map of the current scene; input the 3D feature map of the current scene into the first fully connected layer and the second fully connected layer, and perform feature decoding on the 3D semantic model of the current scene through the first fully connected layer to obtain the feature vector of the vector model in the 3D semantic model of the current scene; among them, the convolutional layer can adopt network structures such as VGGNet and ResNet, and use 3D convolution to replace the 2D convolution operation of the original network structure, so as to extract higher-quality 3D features; the convolutional layer can be regarded as an encoder, and the first fully connected layer can be regarded as a decoder; input the feature vector of the vector model in the 3D semantic model of the current scene into the classifier, and the classifier outputs the category of the vector model; obtain the parameters of the vector model in the 3D semantic model of the current scene through the second fully connected layer, and then obtain the 3D vector grid of the current scene. The 3D vector network includes the category and parameters of the vector model in the 3D semantic model of the current scene.

[0101] In a feasible example, as Figure 7As shown in the figure, the multi-task neural network includes two convolutional layers, two fully connected layers, and a classifier. Among them, the two convolutional layers are the first convolutional layer and the second convolutional layer respectively, and the two fully connected layers include the first fully connected layer and the second fully connected layer; the output data of the first convolutional layer and the output data of the second convolutional layer are the input data of the first fully connected layer and the input data of the second fully connected layer respectively, and the output data of the first fully connected layer is the input data of the classifier; among them, the structure and parameters of the first convolutional layer are the same as those of the second convolutional layer. In other words, the parameters of the first convolutional layer and the second convolutional layer are shared, and the functions of the first convolutional layer and the second convolutional layer are both the same as those of the convolutional layer in Figure 6 ; Figure 7 The functions of the first fully connected layer, the second fully connected layer, and the classifier shown in Figure 6 are the same as the functions of the first fully connected layer, the second fully connected layer, and the classifier shown in

[0102] After obtaining the category and parameters of the vector model in the current scene in the above manner, the vector model in the current scene is obtained from the vector model library according to the category and parameters of the vector model in the current scene, and then the 3D vector grid of the current scene is obtained.

[0103] It can be seen that in the solution of this application, by combining shape semantic segmentation, the problem that vector modeling can only reconstruct plane information is solved, and at the same time, the problem that existing vector modeling cannot adapt to complex indoor scenes is also solved; through the multi-task neural network, the expansion ability of vector modeling is enhanced, enabling a variety of vector models to be reconstructed relying on the 3D semantic model, and then making it possible to obtain a comprehensive and accurate three-dimensional vector grid, truly opening up the path between the 3D semantic model and the 3D vector grid, enabling the system to adapt to various scenes, such as indoor, outdoor, planar, and curved surfaces, etc., greatly improving the applicable range and commercial value of vector modeling. Using the solution of the present invention, the 3D vector grid of indoor and outdoor scenes can be automatically generated, greatly reducing costs, and playing an important role in virtual object placement, scene interaction collision, virtual-real occlusion, etc. in the fields of augmented reality (AR) and virtual reality (VR).

[0104] See Figure 8 , Figure 8 which is a schematic structural diagram of a server provided by an embodiment of the present invention. As Figure 8 shown, the server 800 includes:

[0105] An acquisition unit 801, configured to acquire multiple first images for the current scene, and the multiple first images are consecutive frames;

[0106] A processing unit 802, configured to perform point cloud reconstruction on multiple first images to obtain the point cloud of the current scene and the poses of each of the multiple first images in the multiple first images; perform semantic segmentation on each of the multiple first images to obtain the 2D semantic information of the first image;

[0107] A generating unit 803, configured to generate a 3D semantic model of the current scene according to the point cloud of the current scene, the poses of each of the multiple first images in the multiple first images, and the 2D semantic information of each of the multiple first images, where the 3D semantic model of the current scene is a 3D point cloud carrying category information;

[0108] A feature extraction and classification unit 804, configured to perform feature extraction and classification of a vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene;

[0109] An obtaining unit 801 is further configured to obtain a 3D vector grid of the current scene according to the category and parameters of the vector model in the current scene.

[0110] In a feasible embodiment, the feature extraction and classification unit 804 is specifically configured to:

[0111] Perform feature extraction on the 3D semantic model of the current scene to obtain a 3D feature map of the current scene; perform feature decoding on the 3D feature map of the current scene to obtain a feature vector of the vector model in the 3D semantic model of the current scene; classify the vector model according to the feature vector of the vector model to obtain the category of the vector model; obtain the parameters of the vector model according to the 3D feature map of the current scene.

[0112] In a feasible embodiment, the feature extraction and classification unit 804 is specifically configured to:

[0113] Call a multi-task neural network, and input the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification to obtain the category and parameters of the vector model in the current scene.

[0114] In a feasible embodiment, the multi-task neural network includes a convolutional layer, a first fully connected layer, a second fully connected layer, and a classifier. Among them, the output data of the convolutional layer is the input data of the first fully connected layer and the input data of the second fully connected layer, and the output data of the first fully connected layer is the input data of the classifier. In terms of inputting the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification to obtain the category and parameters of the vector model in the current scene, the feature extraction and classification unit 804 is specifically configured to:

[0115] Input the 3D semantic model of the current scene into the convolutional layer for feature extraction to obtain the 3D feature map of the current scene; input the 3D feature map of the current scene into the first fully connected layer for feature decoding to obtain the feature vector of the vector model in the 3D semantic model of the current scene; input the feature vector of the vector model in the 3D semantic model of the current scene into the classifier for classification to obtain the category of the vector model in the 3D semantic model of the current scene; input the 3D feature map of the current scene into the second fully connected layer for processing to obtain the parameters of the vector model in the 3D semantic model of the current scene.

[0116] In a feasible embodiment, the convolutional layer includes a first convolutional layer and a second convolutional layer, and the structure and parameters of the first convolutional layer are the same as those of the second convolutional layer; the output data of the first convolutional layer is the input data of the first fully connected network; the output data of the second convolutional layer is the output data of the second fully connected layer; in terms of inputting the 3D semantic model of the current scene into the convolutional layer for feature extraction to obtain the 3D feature map of the current scene, the feature extraction and classification unit 804 is specifically used for:

[0117] Input the 3D semantic model of the current scene into the first convolutional layer for feature extraction to obtain the 3D feature map of the current scene; input the 3D semantic model of the current scene into the second convolutional layer for feature extraction to obtain the 3D feature map of the current scene.

[0118] In a feasible embodiment, the generation unit 803 is specifically used for:

[0119] Project the target point cloud onto each of the multiple first images according to the pose of each first image in the multiple first images to obtain the category corresponding to each 3D point in the target point cloud on the first image; wherein, the target point cloud is the point cloud of the current scene; use the 3D points in the target point cloud as nodes, and use the connection lines between the 3D points and the corresponding categories of the 3D points on the first image as edges to construct a graph model based on the Markov random field (MRF); perform MRF optimization solution on the graph model based on the MRF according to the energy function to obtain the 3D semantic model of the current scene, and each 3D point in the 3D semantic model has a unique category.

[0120] It should be noted that the above units (acquisition unit 801, processing unit 802, generation unit 803, and feature extraction and classification unit 804) are used to execute the relevant steps of the above method. For example, the acquisition unit 801 is used to execute the relevant content of S501 and S505, the processing unit 802 is used to execute the relevant content of S502 and S503, the generation unit 803 is used to execute the relevant content of S504, and the feature extraction and classification unit 804 is used to execute the relevant content of S505.

[0121] In this embodiment, the server 800 is presented in the form of a unit. Here, the "unit" may refer to an application-specific integrated circuit (ASIC), a processor and a memory that execute one or more software or firmware programs, an integrated logic circuit, and / or other devices that can provide the above functions. In addition, the above acquisition unit 801, processing unit 802, generation unit 803, and feature extraction and classification unit 804 can be implemented by Figure 9 the processor 901 of the server shown.

[0122] Such as Figure 9 the server 900 shown can be implemented in the Figure 9 structure shown. The server 900 includes at least one processor 901, at least one memory 902, and at least one communication interface 903. The processor 901, the memory 902, and the communication interface 903 are connected through the communication bus and complete communication with each other.

[0123] The processor 901 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the above solution programs.

[0124] The communication interface 1103 is used to communicate with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (Wireless Local Area Networks, WLAN), etc.

[0125] The memory 1102 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or can also be an Electrically Erasable Programmable Read-Only Memory (EEPROM), a Compact Disc Read-Only Memory (CD-ROM) or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory can exist independently and be connected to the processor through a bus. The memory can also be integrated with the processor.

[0126] Among them, the memory 1102 is used to store the application program code for executing the above solution, and is controlled by the processor 501 to execute. The processor 1101 is used to execute the application program code stored in the memory 1102.

[0127] The code stored in the memory 1102 can execute any one of the 3D vector grid generation methods provided above, such as:

[0128] Obtain multiple first images for the current scene, and the multiple first images are consecutive frames; perform point cloud reconstruction on the multiple first images to obtain the point cloud of the current scene and the pose of each of the multiple first images; perform semantic segmentation on each of the multiple first images to obtain the 2D semantic information of the first image; generate a 3D semantic model of the current scene according to the point cloud of the current scene, the pose of each of the multiple first images, and the 2D semantic information of each of the multiple first images, and the 3D semantic model of the current scene is a 3D point cloud carrying category information; perform feature extraction and classification of the vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene; obtain the 3D vector grid of the current scene according to the category and parameters of the vector model in the current scene.

[0129] The embodiment of the present invention also provides a computer storage medium, wherein the computer storage medium can store a program, and when the program is executed, it includes some or all of the steps of any one of the 3D vector grid generation methods recorded in the above method embodiments.

[0130] It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0131] In the above embodiments, the descriptions of the respective embodiments have their own focuses. For the parts not detailed in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0132] In several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division, and there can be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection to each other can be through some interfaces, and the indirect coupling or communication connection of the device or unit can be in an electrical or other form.

[0133] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they can be located in one place, or can be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, in each embodiment of the present invention, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0135] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned memory includes: various media such as USB flash drives, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), external hard drives, magnetic disks, or optical discs that can store program codes.

[0136] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program. This program can be stored in a computer-readable memory, and the memory can include: USB flash drives, read-only memories (English: Read-Only Memory, abbreviated: ROM), random access memories (English: Random Access Memory, abbreviated: RAM), magnetic disks, or optical discs, etc.

[0137] The above has introduced the embodiments of the present invention in detail. Specific examples are used in this article to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A 3D vector grid generation method, characterized in that, Including: Obtaining multiple first images for the current scene, where the multiple first images are consecutive frames; Performing point cloud reconstruction on the multiple first images to obtain the point cloud of the current scene and the pose of each first image in the multiple first images; Performing semantic segmentation on each first image in the multiple first images to obtain the 2D semantic information of each first image; Generating a 3D semantic model of the current scene according to the point cloud of the current scene, the pose of each first image in the multiple first images, and the 2D semantic information of each first image in the multiple first images, where the 3D semantic model of the current scene is a 3D point cloud carrying category information; Performing feature extraction and classification of the vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene; Obtaining the vector model in the current scene according to the category and parameters of the vector model in the current scene; The generating the 3D semantic model of the current scene according to the point cloud of the current scene, the pose of each first image in the multiple first images, and the 2D semantic information of each first image in the multiple first images includes: Projecting the target point cloud onto each first image according to the pose of each first image in the multiple first images to obtain the category corresponding to each 3D point in the target point cloud on each first image; where the target point cloud is the point cloud of the current scene; Taking the 3D points in the target point cloud as nodes and taking the connection line between the 3D point and the corresponding category of the 3D point in the first image as edges to construct a graph model based on the Markov random field (MRF); Performing MRF optimization solution on the graph model based on MRF according to the energy function to obtain the 3D semantic model of the current scene, where each 3D point in the 3D semantic model has a unique category.

2. The method according to claim 1, characterized in that, The performing feature extraction and classification of the vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene includes: Performing feature extraction on the 3D semantic model of the current scene to obtain a 3D feature map of the current scene; Performing feature decoding on the 3D feature map of the current scene to obtain the feature vector of the vector model in the 3D semantic model of the current scene; Classifying the vector model according to the feature vector of the vector model to obtain the category of the vector model; Obtaining the parameters of the vector model according to the 3D feature map of the current scene.

3. The method according to claim 2, wherein The performing feature extraction and classification of the vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene includes: Invoking a multi-task neural network, inputting the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification to obtain the category and parameters of the vector model in the current scene.

4. The method according to claim 3, wherein The multi-task neural network includes a convolutional layer, a first fully connected layer, a second fully connected layer, and a classifier. Among them, the output data of the convolutional layer is the input data of the first fully connected layer and the input data of the second fully connected layer. The output data of the first fully connected layer is the input data of the classifier. Inputting the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification to obtain the category and parameters of the vector model in the current scene includes: Input the 3D semantic model of the current scene into the convolutional layer for feature extraction to obtain the 3D feature map of the current scene; Input the 3D feature map of the current scene into the first fully connected layer for feature decoding to obtain the feature vector of the vector model in the 3D semantic model of the current scene; Input the feature vector of the vector model in the 3D semantic model of the current scene into the classifier for classification to obtain the category of the vector model in the 3D semantic model of the current scene; Input the 3D feature map of the current scene into the second fully connected layer for processing to obtain the parameters of the vector model in the 3D semantic model of the current scene.

5. The method according to claim 4, wherein The convolutional layer includes a first convolutional layer and a second convolutional layer, and the structure and parameters of the first convolutional layer are the same as those of the second convolutional layer; The output data of the first convolutional layer is the input data of the first fully connected network; The output data of the second convolutional layer is the output data of the second fully connected layer; Input the 3D semantic model of the current scene into the first convolutional layer for feature extraction to obtain the 3D feature map of the current scene; Input the 3D semantic model of the current scene into the second convolutional layer for feature extraction to obtain the 3D feature map of the current scene.

6. A server, characterized in that, Including: An acquisition unit for acquiring multiple first images for the current scene, and the multiple first images are consecutive frames; A processing unit for performing point cloud reconstruction on the multiple first images to obtain the point cloud of the current scene and the pose of each of the multiple first images; Performing semantic segmentation on each of the multiple first images to obtain the 2D semantic information of each of the multiple first images; A generation unit for generating the 3D semantic model of the current scene according to the point cloud of the current scene, the pose of each of the multiple first images, and the 2D semantic information of each of the multiple first images. The 3D semantic model of the current scene is a 3D point cloud carrying category information; A feature extraction and classification unit for performing feature extraction and classification of the vector model on the 3D semantic model of the current scene to obtain the category and parameters of the vector model in the current scene; The acquisition unit is further configured to acquire the 3D vector grid in the current scene according to the category and parameters of the vector model in the current scene; The generation unit is specifically configured to: Project the target point cloud onto each of the first images according to the pose of each first image in the multiple first images, to obtain the category corresponding to each 3D point in the target point cloud on each of the first images; wherein, the target point cloud is the point cloud of the current scene. Use the 3D points in the target point cloud as nodes, and use the connection lines between the 3D points and the categories corresponding to the 3D points in the first image as edges to construct a graph model based on the Markov Random Field (MRF). Perform MRF optimization and solution on the graph model based on MRF according to the energy function to obtain the 3D semantic model of the current scene, and each 3D point in the 3D semantic model has a unique category.

7. The server according to claim 6, wherein The feature extraction and classification unit is specifically used for: Extract features from the 3D semantic model of the current scene to obtain the 3D feature map of the current scene. Decode the features of the 3D feature map of the current scene to obtain the feature vectors of the vector model in the 3D semantic model of the current scene. Classify the vector model according to the feature vectors of the vector model to obtain the category of the vector model. Obtain the parameters of the vector model according to the 3D feature map of the current scene.

8. The server according to claim 7, characterized in that, The feature extraction and classification unit is specifically used for: Call a multi-task neural network, input the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification, to obtain the category and parameters of the vector model in the current scene.

9. The server according to claim 8, characterized in that, The multi-task neural network includes a convolutional layer, a first fully connected layer, a second fully connected layer, and a classifier. Among them, the output data of the convolutional layer is the input data of the first fully connected layer and the input data of the second fully connected layer, and the output data of the first fully connected layer is the input data of the classifier. In the aspect of inputting the 3D semantic model of the current scene into the multi-task neural network for feature extraction and classification to obtain the category and parameters of the vector model in the current scene, the feature extraction and classification unit is specifically used for: Input the 3D semantic model of the current scene into the convolutional layer for feature extraction to obtain the 3D feature map of the current scene. Input the 3D feature map of the current scene into the first fully connected layer for feature decoding to obtain the feature vectors of the vector model in the 3D semantic model of the current scene. Input the feature vectors of the vector model in the 3D semantic model of the current scene into the classifier for classification to obtain the category of the vector model in the 3D semantic model of the current scene. Input the 3D feature map of the current scene into the second fully connected layer for processing to obtain the parameters of the vector model in the 3D semantic model of the current scene.

10. The server according to claim 9, wherein The convolutional layer includes a first convolutional layer and a second convolutional layer, and the structure and parameters of the first convolutional layer are the same as those of the second convolutional layer. The output data of the first convolutional layer is the input data of the first fully connected network. The output data of the second convolutional layer is the output data of the second fully connected layer. In the aspect of inputting the 3D semantic model of the current scene into the convolutional layer for feature extraction to obtain the 3D feature map of the current scene, the feature extraction and classification unit is specifically configured to: Input the 3D semantic model of the current scene into the first convolutional layer for feature extraction to obtain the 3D feature map of the current scene; Input the 3D semantic model of the current scene into the second convolutional layer for feature extraction to obtain the 3D feature map of the current scene.

11. A server, characterized in that, Comprising: A memory for storing instructions; And A processor coupled to the memory; Wherein, when the processor executes the instructions, the method according to any one of claims 1-5 is executed.

12. A chip system, characterized in that, The chip system is applied to an electronic device; the chip system includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected by lines; the interface circuits are configured to receive signals from the memory of the electronic device and send the signals to the processors, and the signals include computer instructions stored in the memory; when the processors execute the computer instructions, the electronic device executes the method according to any one of claims 1-5.

13. A computer storage medium, characterized in that, The computer storage medium stores a computer program, the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1-5.

14. A computer program product, characterized in that, Including computer instructions, when the computer instructions run on an electronic device, the electronic device is caused to execute the method according to any one of claims 1-5.

Images