Method, device, equipment and medium for determining connection relationship

Abstract

The present application discloses a method, device, equipment and medium for determining connection relationships, which belongs to the field of three-dimensional reconstruction. The method includes: obtaining line cloud data and a connection point set for reconstructing a three-dimensional model, the line cloud data includes multiple three-dimensional line segments related to the three-dimensional model, the connection points in the connection point set are the contour points of the three-dimensional model, and the three-dimensional line segments refer to the line segments in the three-dimensional space; constructing a target feature vector based on the connection point pairs constructed from any two connection points in the connection point set, and the distance between the connection points in the connection point pair and the three-dimensional line segments in the line cloud data; inputting the target feature vector into the prediction network of the connection relationship; and obtaining the connection relationship between any two connection points in the connection point set in the three-dimensional model through the prediction network. The above scheme overcomes the problems in related technologies that point cloud data is difficult to obtain and the generalization of the connection point estimation network is poor.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction, and in particular to a method, device, equipment and medium for determining a connection relationship. Background Art

[0002] 3D reconstruction involves creating a 3D model of a 3D object suitable for computer representation and processing. During the 3D reconstruction process, the 3D model is constructed based on the connection points of the 3D model and the connections between them. The connection points are the contour points of the 3D model.

[0003] In the related art, the point cloud data generated by the CAD model of the three-dimensional object is first obtained, and the features of the point cloud data are obtained through the feature extraction network. Then, the obtained point cloud data is sampled to obtain multiple point blocks (patches), and each point block is input into the connection point detection network to determine whether there is a connection point in the current point block. The point block determined to have a connection point is then input into the connection point locator to locate the position of the connection point. The located connection points are combined in pairs to obtain multiple candidate line segments. Then, multiple points are uniformly sampled between the two endpoints of the candidate line segment, and then the coordinates of the sampled points are interpolated in the features of the above-mentioned point cloud data to obtain the feature vector of the sampled point. The feature vectors of the multiple sampled points of the candidate line segment are input into the classifier to obtain the probability that the endpoints of the candidate line segment are connected in the constructed three-dimensional model.

[0004] Point cloud data in related technologies needs to be extracted from CAD models. However, the accuracy of CAD models built by the scene front-end is often limited, resulting in a significant gap between the point cloud data extracted from the CAD model and the point cloud data used to train the connection estimation network. This in turn leads to poor generalization performance of the trained connection estimation network. When switching from a high-precision CAD model to a lower-precision CAD model, the connection estimation network cannot guarantee the accuracy of the output connection relationships. Summary of the Invention

[0005] This application provides a method, device, equipment, and medium for determining connection relationships, which can overcome the problems in related technologies such as the difficulty in obtaining point cloud data and the poor generalization of the estimated network of connection points. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for determining a connection relationship is provided, the method comprising:

[0007] Obtain line cloud data and a connection point set for reconstructing a 3D model. The line cloud data includes multiple 3D line segments related to the 3D model. The connection points in the connection point set are contour points of the 3D model. The 3D line segments refer to line segments in 3D space.

[0008] Construct a target feature vector based on a connection point pair constructed from any two connection points in the connection point set and the distance between the connection points in the connection point pair and the three-dimensional line segment in the line cloud data;

[0009] Input the target feature vector into the prediction network of the connection relationship;

[0010] By predicting the network, the connection relationship between any two connection points in the connection point set in the three-dimensional model is obtained.

[0011] According to another aspect of the present application, a method for training a connection relationship prediction network is provided, the method comprising:

[0012] Obtaining sample line cloud data and a sample connection point set for reconstructing a sample three-dimensional model, wherein the sample line cloud data includes a plurality of sample three-dimensional line segments associated with the sample three-dimensional model, the sample connection points in the sample connection point set are contour points of the sample three-dimensional model, and the sample three-dimensional line segments refer to line segments in three-dimensional space;

[0013] Construct a target sample feature vector based on a sample connection point pair constructed from any two sample connection points in the sample connection point set, and the distance between the sample connection points in the sample connection point pair and the sample three-dimensional line segment in the sample line cloud data;

[0014] Input the target sample feature vector into the prediction network;

[0015] Predicting a first connection relationship corresponding to the sample connection point set through a prediction network, where the first connection relationship includes a connection relationship between any two connection points in the predicted sample connection point set on the sample three-dimensional model;

[0016] The prediction network is trained based on the second connection relationship and the first connection relationship of the sample connection point set, where the second connection relationship includes the connection relationship between any two connection points in the sample connection point set obtained by labeling.

[0017] According to another aspect of the present application, a training device for a connection relationship prediction network is provided, the device comprising:

[0018] an acquisition module, configured to acquire sample line cloud data and a sample connection point set for reconstructing a sample three-dimensional model, wherein the sample line cloud data includes a plurality of sample three-dimensional line segments associated with the sample three-dimensional model, the sample connection points in the sample connection point set are contour points of the sample three-dimensional model, and the sample three-dimensional line segments refer to line segments in three-dimensional space;

[0019] A construction module is used to construct a target sample feature vector based on a sample connection point pair constructed from any two sample connection points in the sample connection point set, and the distance between the sample connection points in the sample connection point pair and the sample three-dimensional line segment in the sample line cloud data;

[0020] Input module, used to input the target sample feature vector into the prediction network;

[0021] A prediction module, configured to predict, through a prediction network, a first connection relationship corresponding to the sample connection point set, the first connection relationship including a connection relationship between any two sample connection points in the predicted sample connection point set on the sample three-dimensional model;

[0022] The training module is used to train the prediction network according to the second connection relationship and the first connection relationship of the sample connection point set, where the second connection relationship includes the connection relationship between any two sample connection points in the sample connection point set obtained by labeling.

[0023] According to another aspect of the present application, a device for determining a connection relationship is provided, the device comprising:

[0024] An acquisition module, configured to acquire line cloud data and a connection point set for reconstructing a three-dimensional model, wherein the line cloud data includes a plurality of three-dimensional line segments associated with the three-dimensional model, the connection points in the connection point set are contour points of the three-dimensional model, and the three-dimensional line segments refer to line segments in three-dimensional space;

[0025] A construction module is used to construct a target feature vector based on a connection point pair constructed from any two connection points in the connection point set and the distance between the connection points in the connection point pair and the three-dimensional line segment in the line cloud data;

[0026] An input module, used to input the target feature vector into the prediction network of the connection relationship;

[0027] The prediction module is used to obtain the connection relationship between any two connection points in the connection point set in the three-dimensional model through the prediction network.

[0028] According to one aspect of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the method for determining the connection relationship as described above, or the method for training a network for predicting the connection relationship.

[0029] According to another aspect of the present application, a computer-readable storage medium is provided, which stores a computer program, and the computer program is loaded and executed by a processor to implement the method for determining the connection relationship as described above, or the method for training a network for predicting the connection relationship.

[0030] According to another aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described method for determining a connection relationship or the method for training a network for predicting a connection relationship.

[0031] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0032] Based on the connection point pairs constructed from any two connection points in the connection point set, and the distances between the connection points in the connection point pairs and the three-dimensional line segments in the line cloud data, a target feature vector for predicting connection relationships is constructed. This provides a method for predicting connection relationships based on line cloud data and connection points, overcoming the problems in related technologies of difficulty in obtaining point cloud data and poor generalization of the connection point estimation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application;

[0035] Figure 2 is a flow chart of a method for determining a connection relationship provided by an exemplary embodiment of the present application;

[0036] Figure 3 is a schematic diagram of a connection point provided by an exemplary embodiment of the present application;

[0037] Figure 4 is a schematic diagram of line cloud data provided by another exemplary embodiment of the present application;

[0038] Figure 5 is a schematic diagram of a connection relationship determination architecture provided by another exemplary embodiment of the present application;

[0039] Figure 6 is a schematic diagram of a predicted network of connection relationships provided by an exemplary embodiment of the present application;

[0040] Figure 7 is a flowchart of a method for training a connection relationship prediction network provided by an exemplary embodiment of the present application;

[0041] Figure 8 This is a structural block diagram of a device for determining a connection relationship provided by an exemplary embodiment of the present application;

[0042] Figure 9 This is a structural block diagram of a training device for a connection relationship prediction network provided by an exemplary embodiment of the present application;

[0043] Figure 10 It is a structural block diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0045] First, a brief introduction to the terms involved in the embodiments of this application is given:

[0046] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0047] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0048] 3D reconstruction refers to the creation of a 3D model of a real-world object suitable for computer representation and processing. The most crucial step in this process is determining the connection relationships between connection points. The above background technology describes methods for determining connection relationships based on point cloud data. The following describes methods for determining connection relationships based on 2D images.

[0049] In the related art, a two-dimensional image of a three-dimensional object is first input into a deep convolutional neural network to obtain a high-dimensional feature map of the two-dimensional image. For example, the high-dimensional feature map is represented as 256*256*M, where 256*256 represents the coordinate dimension of the two-dimensional image and M represents the feature dimension of the feature map. Afterwards, the two-dimensional image is input into a connection point prediction network to obtain the two-dimensional coordinates of the connection points. Then, multiple connection points are combined in pairs to obtain multiple two-dimensional line segments. Uniform sampling is performed between the two endpoints of each two-dimensional line segment to obtain multiple sampling points. Bilinear interpolation is performed in the high-dimensional feature map based on the coordinates of the multiple sampling points to obtain the feature vector of the sampling point. For example, the two-dimensional coordinates of the sampling point are (0.5, 0.5). The feature vectors of four points with coordinates (0, 0), (0, 1), (1, 0), and (1, 1) are found in the high-dimensional feature map. The feature vector of the sampling point (0.5, 0.5) is obtained by averaging the feature vectors of these four points. Finally, the feature vectors of multiple sampling points of the two-dimensional line segment are input into the connection relationship prediction network to predict the probability that the two endpoints of the two-dimensional line segment are actually connected.

[0050] However, after using the aforementioned "determining connection relationships based on 2D images" method, the connection points and connection relationships still need to be mapped from 2D space to 3D space. However, this mapping step to 3D space presents the following problems: 1. Currently, there is only a method for mapping connection points in 2D space to connection points in 3D space, and there is no way to map connection relationships in 2D space to connection relationships in 3D space. 2. If the connection relationships in 2D space are directly inherited from those in 3D space, the resulting 3D model will not be a clean wireframe representation. The reconstructed 3D model will be affected by incorrect 2D connection relationships and redundant 3D connection points. Furthermore, incorrect mapping relationships when mapping 2D connection points to 3D space will also have serious consequences.

[0051] The method for determining connection points proposed in this application can overcome the defects of the two methods for determining connection relationships in the above-mentioned related technologies.

[0052] Figure 1 A schematic diagram of a computer system provided by an exemplary embodiment of the present application is shown.

[0053] In one embodiment, the method for determining a connection relationship provided in the present application is performed by at least one server 110. In one embodiment, the method for determining a connection relationship provided in the present application is performed by at least one terminal 120. In one embodiment, the method for determining a connection relationship provided in the present application is performed collaboratively by at least one terminal 120 and at least one server 110. Figure 1 Only one server and one terminal are shown.

[0054] Optionally, server 110 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0055] Optionally, the terminal 120 may be a smart phone, a wearable device, a smart TV, a vehicle-mounted terminal, a tablet computer, a laptop computer, a desktop computer, a smart speaker, etc., but is not limited thereto.

[0056] Optionally, the terminal 120 and the server 110 may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0057] Figure 2 A flowchart showing a method for determining a connection relationship provided by an exemplary embodiment of the present application is shown. Figure 1 The server 110 shown is executed for example, and the method includes:

[0058] Step 210: obtaining line cloud data and a set of connection points for reconstructing a three-dimensional model;

[0059] Three-dimensional model: refers to a model obtained by three-dimensional reconstruction using the connection relationship determination method provided in this application.

[0060] Connection point set: In the present application, the connection point set includes all the connection points for constructing a three-dimensional model. The connection points are the contour points of the three-dimensional model, and all the contour points of the three-dimensional model constitute the point frame structure of the three-dimensional model. In one embodiment, the connection point set can be obtained based on the contour points of the three-dimensional object. Optionally, the contour points of the three-dimensional object used for three-dimensional reconstruction are directly used as connection points. For example, the contour points of a house in the real world are used as connection points. Optionally, the connection points are obtained by sampling around the contour points of the three-dimensional object used for three-dimensional reconstruction. For example, the coordinates of the contour points of a house in the real world are (1, 1, 1.05), and the coordinates of the connection points sampled around the contour points are (1, 1, 1). In another embodiment, the connection point set can be predicted by a neural network model. Optionally, the image obtained by shooting the three-dimensional object is input into the connection point prediction network, and the connection points in the image can be determined by the connection point prediction network.

[0061] Indicative, Figure 3 A connection point set provided by an exemplary embodiment of the present application is shown. Figure 3The bold black dots in part (A) are the connection points. Figure 3 All the bold black dots in part (A) constitute a set of connection points; Figure 3 The bold black dots in part (B) are the connection points. Figure 3 All the bold black dots in part (B) constitute a set of connection points; Figure 3 The bold black dots in part (C) are the connection points. Figure 3 All the bold black dots in part (C) constitute a set of connection points; Figure 3 The bold black dots in part (D) are the connection points. Figure 3 All bold black dots in part (D) constitute a connection point set. Figure 3 How the bold black dots are connected is the connection relationship to be determined in this application.

[0062] Line cloud data: In this application, it refers to multiple three-dimensional line segments associated with a three-dimensional model. A three-dimensional line segment refers to a line segment in three-dimensional space. In one embodiment, the acquisition method of line cloud data includes the following steps 1 to 4:

[0063] 1. Obtain a multi-view image set of a three-dimensional object corresponding to the three-dimensional model. The multi-view image set includes multiple images of the three-dimensional object obtained by photographing the three-dimensional object from multiple perspectives. Schematically, the three-dimensional object is a house, and the multi-view image set of the three-dimensional object includes multiple images of the house obtained by photographing the house from multiple perspectives.

[0064] 2. Detecting multiple two-dimensional line segments from a multi-view image set; illustratively, based on at least one of the image's color information, depth information, and texture information, multiple two-dimensional line segments are detected from multiple images, where a two-dimensional line segment refers to a line segment in a two-dimensional space.

[0065] For example, for pixel A in a two-dimensional image, if the difference between the pixel values of pixel A and pixel B is less than a threshold, pixel A and pixel B are connected in series, with pixel B being the adjacent pixel of pixel A. Then, if the difference between the pixel values of pixel B and pixel C is less than a threshold, pixel B and pixel C are connected in series, with pixel C being the adjacent pixel of pixel B. This process continues in this way until the difference between the pixel value of the last pixel and the pixel value of its adjacent pixels is no less than the threshold, at which point the connection operation is stopped, and a complete two-dimensional line segment is finally obtained.

[0066] 3. According to the internal and external parameters of the multi-view image set, multiple two-dimensional line segments are converted into multiple three-dimensional line segments; illustratively, according to the internal and external parameters of the multiple images, the multiple images are set in the same three-dimensional space, and the multiple two-dimensional line segments corresponding to the multiple images are converted into multiple three-dimensional line segments in the same three-dimensional space.

[0067] 4. Determine multiple 3D line segments as line cloud data. Schematically, Figure 4 Figure 2 shows line cloud data provided by an exemplary embodiment of the present application. Optionally, after obtaining multiple 3D line segments, the segments are normalized. For any 3D line segment, its 3D coordinate is normalized by subtracting the mean of the 3D coordinates of all 3D line segments from its 3D coordinates, and then dividing by the maximum value of the L2 norm of the 3D coordinates of all 3D line segments. The 3D coordinates are the x-axis coordinates, y-axis coordinates, or z-axis coordinates.

[0068] Schematically, the normalized formula is expressed as: X'=(XX average ) / X max ; Where X' is the 3D coordinate of the 3D line segment after normalization, X is the 3D coordinate of the 3D line segment before normalization, X average is the mean of the three-dimensional coordinates of all three-dimensional line segments before normalization, X max It is the maximum value of the L2 norm of the three-dimensional coordinates of all three-dimensional line segments before normalization.

[0069] Step 220 , constructing a target feature vector based on a connection point pair constructed from any two connection points in the connection point set and the distance between the connection points in the connection point pair and the three-dimensional line segment in the line cloud data;

[0070] In one embodiment, the process of constructing the target feature vector includes the following steps 1 to 4:

[0071] 1. For the nth connection point in the connection point set, determine L three-dimensional line segments from the line cloud data whose distance to the nth connection point is less than the distance threshold. The nth connection point is any connection point in the connection point set. For example, the distance threshold is 20, L is a fixed value of 8, and the 8 three-dimensional line segments in the line cloud data whose distance to the nth connection point is not greater than 20 and which are closest to the nth connection point are determined. It should be noted that when 8 three-dimensional line segments cannot be obtained within the range of no more than 20 from the nth connection point, the empty positions are padded with 0 to satisfy the form of the first eigenvector below. n and L are positive integers.

[0072] 2. Based on the N connection points in the connection point set, the L three-dimensional line segments whose distance from the nth connection point is less than a distance threshold, and the line segment features of the lth three-dimensional line segment in the L three-dimensional line segments, obtain a first feature vector represented as N×L×K. The lth three-dimensional line segment is any one of the L three-dimensional line segments, and K is the number of features of the lth three-dimensional line segment. Optionally, the value of K is 7, and the features of the lth three-dimensional line segment include the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the first endpoint of the lth three-dimensional line segment, the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the second endpoint, and the distance between the lth three-dimensional line segment and the nth connection point. L, l, and K are all positive integers.

[0073] 3. For the nth connection point, pair the connection points in the connection point set except the nth connection point with the nth connection point. The connection point set is paired to obtain A pair of connection points.

[0074] 4. Transform the first eigenvector represented as N×L×K into The second eigenvector is determined as the target eigenvector.

[0075] Optionally, the first eigenvector is represented as a 256×8×7 tensor, and the second eigenvector is represented as Tensor of .

[0076] Step 230, inputting the target feature vector into the prediction network of the connection relationship;

[0077] Among them, the prediction network of the connection relationship is a neural network.

[0078] In one embodiment, it is represented as The second feature vector is input to the prediction network of the connection relationship.

[0079] Step 240: Obtain the connection relationship between any two connection points in the connection point set in the three-dimensional model through the prediction network.

[0080] In one embodiment, the connection relationship between any two connection points in the connection point set in the three-dimensional model is predicted using a connection relationship prediction network, wherein the connection relationship prediction network is a neural network for predicting connection relationships.

[0081] In summary, based on the connection point pairs constructed from any two connection points in the connection point set, and the distances between the connection points in the connection point pairs and the three-dimensional line segments in the line cloud data, a target feature vector for predicting connection relationships is constructed. This provides a method for predicting connection relationships based on line cloud data and connection points, overcoming the problems in related technologies of difficulty in obtaining point cloud data and poor generalization of the connection point estimation network.

[0082] Furthermore, a specific method for acquiring line cloud data and a representation of target feature vectors are provided, thereby ensuring the feasibility of the method for determining the connection relationship provided in this application.

[0083] based on Figure 2 In the optional embodiment shown, after the second feature vector is constructed in step 220, at least one of the following steps may be further included:

[0084] (1) Input the second eigenvector into the multilayer perceptron, and get the expression The third eigenvector of , the value of K' is greater than K, K' is a positive integer;

[0085] Schematically, the second eigenvector is expressed as The third eigenvector is expressed as The multilayer perceptron is used to improve the expressive power of the second eigenvector in the feature dimension of the three-dimensional line segment.

[0086] (2) Input the second eigenvector into the first transform Transformer network, and get the representation as The fourth eigenvector of L×K;

[0087] Among them, the first Transformer network is used to add a self-attention mechanism in the number dimension of three-dimensional line segments.

[0088] The first Transformer network is used to add a self-attention mechanism in the quantity dimension of the three-dimensional line segments. It can be understood that the first Transformer network adds a self-attention mechanism within each group of three-dimensional line segments.

[0089] (3) Input the second eigenvector into the pooling network and get the representation as The fifth eigenvector of ;

[0090] Optionally, a pooling network is used to perform average pooling in the dimension of the number of 3D line segments. The output of the pooling network is expressed as The fifth eigenvector of . Schematically, the second eigenvector is expressed as The fifth eigenvector is expressed as Average pooling is used to reduce the dimension of the feature vector and avoid overfitting of the final network.

[0091] (4) Input the second eigenvector into the second Transformer network, and get the representation: The sixth eigenvector of ;

[0092] Among them, the second Transformer network is used to add a self-attention mechanism in the dimension of the number of connection point pairs.

[0093] The second Transformer network is used to add a self-attention mechanism in the dimension of the number of connection point pairs. It can be understood that the second Transformer network adds a self-attention mechanism between multiple groups of three-dimensional line segments.

[0094] Figure 5 A schematic diagram of a connection method determination architecture provided by an exemplary embodiment of the present application is shown. Figure 1The server execution shown is illustrated for illustrative purposes, and the architecture includes a nearest segment sampling network 510 , a combination network 520 , a multi-layer perceptron 530 , a first Transformer network 540 , a pooling network 550 , a second Transformer network 560 , and a prediction network 570 .

[0095] Nearest line segment sampling network 510:

[0096] In one embodiment, the connection point set and line cloud data are input into the nearest line segment sampling network 510, which outputs a feature vector represented as N×L×K.

[0097] In one embodiment, for the nth connection point in the connection point set, L three-dimensional line segments are determined from the line cloud data whose distance from the nth connection point is less than a distance threshold; the nth connection point is any connection point in the connection point set. For example, if the distance threshold is 20 and L is a fixed value of 8, the eight three-dimensional line segments closest to the nth connection point are determined from the line cloud data, whose distance from the nth connection point is no greater than 20.

[0098] According to the N connection points in the connection point set, L three-dimensional line segments whose distance from the nth connection point is less than the distance threshold, and the segment features of the lth three-dimensional line segment in the L three-dimensional line segments, a first feature vector represented as N×L×K is obtained.

[0099] In one embodiment, the value of K is 7, and the characteristics of the first three-dimensional line segment include the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the first endpoint of the first three-dimensional line segment, the x-axis coordinate, y-axis coordinate, and z-axis coordinate of the second endpoint of the first three-dimensional line segment, and the distance between the first three-dimensional line segment and the nth connection point. Optionally, the coordinates of the first endpoint and the second endpoint are both obtained by subtracting the coordinate of the nth connection point, which is intended to achieve decentralization.

[0100] Exemplarily, the feature vector output by the nearest line segment sampling network 510 is represented as a 256×8×7 tensor.

[0101] It should be noted that when 8 three-dimensional line segments cannot be obtained within a range of no more than 20 from the nth connection point, the empty positions are filled with 0 to satisfy the form of the N×L×K eigenvector.

[0102] For example, the distance between the nth connection point and the three-dimensional line segment can be calculated using the following formula:

[0103]

[0104] Among them, the two endpoints of the three-dimensional line segment are A and B, the nth connecting point is point P, and d is the distance.

[0105] Combinatorial Networks 520:

[0106] In one embodiment, the combining network 520 transforms the feature vector represented as N×L×K into a feature vector represented as The eigenvector of .

[0107] In one embodiment, for the nth connection point, the connection points in the connection point set except the nth connection point are paired with the nth connection point; the connection point set is paired to obtain The first eigenvector represented as N×L×K is transformed into The second eigenvector of . Schematically, the second eigenvector is expressed as Tensor of .

[0108] Multilayer Perceptron 530:

[0109] In one embodiment, the multilayer perceptron 530 is represented as The eigenvector of is transformed into The eigenvector of .

[0110] Schematically, the multilayer perceptron 530 is represented as The eigenvector of is transformed into The multi-layer perceptron 530 is used to improve the expression capability of the feature dimension of the three-dimensional line segment.

[0111] In one embodiment, the multilayer perceptron 530 is composed of multiple fully connected layers, BN (Batch Normalization) layers, and nonlinear layers. Optionally, the number of input channels of the multilayer perceptron 530 is 7, and the number of output channels of the final layer is K'. When the number of output channels K' of the final layer is 256, the number of output channels of each layer of the multilayer perceptron 530 can be 32, 64, 128, and 256, respectively.

[0112] First Transformer Network 540:

[0113] In one embodiment, the first Transformer network 540 is represented as The feature vector of is transformed into a feature vector with a self-attention mechanism added in the dimension of the number of three-dimensional line segments. The feature vector is expressed as

[0114] In one embodiment, the first Transformer network 540 includes an encoder self-attention layer, and the first Transformer network 540 is used to add a self-attention mechanism within each set of three-dimensional line segments (i.e., the "2L" dimension).

[0115] Pooling Network 550:

[0116] In one embodiment, the pooling network 550 is represented as The eigenvector of is transformed into The eigenvector of .

[0117] In one embodiment, the pooling network 550 is used to Average pooling is performed on the number dimension of the three-dimensional line segments of the feature vector (i.e., the "2L" dimension, which can be understood as the dimension within each group of three-dimensional line segments), and the output is The eigenvector of .

[0118] Second Transformer Network 560:

[0119] In one embodiment, the second Transformer network 560 is represented as The feature vector of is transformed into a feature vector with a self-attention mechanism added in the dimension of the number of connection point pairs. The feature vector is expressed as

[0120] In one embodiment, the second Transformer network 560 includes a self-attention layer of an encoder, and the second Transformer network 560 is used to dimension) to add a self-attention mechanism.

[0121] Prediction Network 570:

[0122] In one embodiment, the prediction network 570 will The eigenvector of C is the number of categories obtained by classification. The feature vector representation of The probability distribution of connection point pairs in C-type connection relationships. Optionally, C-type connection relationships include:

[0123] The two connection points in a connection point pair belong to the same connection point;

[0124] The two connection points in a connection point pair are directly connected;

[0125] The two connection points in a connection point pair are connected by p connection points, where the value of p is any one from 1 to C-3;

[0126] The two connection points in a connection point pair are connected by at least C-2 connection points.

[0127] For example, if C is 4, then the four types of connection relationships include: the two connection points in a connection point pair belong to the same connection point; the two connection points in a connection point pair are directly connected; the two connection points in a connection point pair are connected through 1 connection point; the two connection points in a connection point pair are connected through at least 2 connection points.

[0128] In one embodiment, the prediction network 570 is composed of at least one classification layer and an output layer stacked in sequence. The classification layer is composed of a fully connected layer, a batch normalization layer, a nonlinear layer, and a dropout layer stacked in sequence; the output layer includes a fully connected layer; wherein the number of output channels of the output layer is C.

[0129] Exemplarily, the prediction network 570 is composed of three classification layers and an output layer stacked in sequence. Figure 6 As shown, the prediction network 570 includes a first classification layer 571, a second classification layer 572, a third classification layer 573 and an output layer 574. The first classification layer 571, the second classification layer 572 and the third classification layer 573 each include a fully connected layer, a BN layer, a nonlinear layer and a Dropout layer.

[0130] Schematically, the feature vector input to the prediction network 570 is represented as The feature vector output by the prediction network 570 is represented as The number of output channels of the first classification layer 571 is 128, the number of output channels of the second classification layer 572 is 64, the number of output channels of the third classification layer 573 is 32, and the number of output channels of the output layer 574 is 4.

[0131] In summary, the above provides an architecture for determining the connection relationship, which is, in order, the nearest line segment sampling network 510, the combination network 520, the multi-layer perceptron 530, the first Transformer network 540, the pooling network 550, the second Transformer network 560 and the prediction network 570, ensuring the feasibility of the method for determining the connection relationship provided in this application.

[0132] In one embodiment, the following Table 1 shows Figure 5 The connection relationship determination framework shown in the figure is an indicator of the effectiveness of predicting connection relationships. In this case, the distance threshold for the 3D line segment of the nth connection point sampled from the line cloud data is 20, and the probability threshold is 0.3. The probability threshold of 0.3 indicates that when the probability of a predicted connection point pair belonging to the "two connection points in the connection point pair are directly connected" category is greater than 0.3, the connection point pair is considered directly connected.

[0133] Table 1

[0134]

[0135] Optionally, for the nth connection point, precision refers to the ratio of the number of actual connected connection points among the predicted connection points directly connected to the nth connection point to the number of predicted connection points. Recall refers to the ratio of the number of actual connected connection points among the predicted connection points directly connected to the nth connection point to the number of actual connected connection points.

[0136] For example, the predicted connection points directly connected to the nth connection point are connection point 1, connection point 2, connection point 3, and connection point 4. In the test data set, the connection points directly connected to the nth connection point have been marked as connection point 1, connection point 2, connection point 5, connection point 6, and connection point 7. The predicted accuracy of the nth connection point is 50% (connection point 1 and connection point 2 are correct, connection point 3 and connection point 4 are incorrect), and the recall rate of the nth connection point is 40% (connection points 1 and 2 are successfully recalled, connection points 5, 6, and 7 are not recalled).

[0137] Optionally, for the overall accuracy of the 3D model, after calculating the accuracy of the nth connection point, all connection points of the 3D reconstructed model are also fully considered, resulting in an accuracy of 27.39% in Table 1. Optionally, for the overall recall of the 3D model, after calculating the recall of the nth connection point, all connection points of the 3D reconstructed model are also fully considered, resulting in a recall of 94.45% in Table 1.

[0138] Optionally, the scores in Table 1 above are weighted according to precision and recall.

[0139] Figure 7 A flowchart of a method for training a connection relationship prediction network is also provided. Figure 1 The server 110 shown is executed for illustration purposes only. The method includes:

[0140] Step 710 , obtaining sample line cloud data and a sample connection point set for reconstructing a sample three-dimensional model;

[0141] The sample line cloud data includes multiple sample three-dimensional line segments related to the sample three-dimensional model, the sample connection points in the sample connection point set are contour points of the sample three-dimensional model, and the sample three-dimensional line segments refer to line segments in three-dimensional space.

[0142] Sample 3D model: refers to the model obtained by 3D reconstruction during training.

[0143] Sample line cloud data: In this application, it refers to multiple three-dimensional line segments associated with the sample three-dimensional model. Three-dimensional line segments refer to line segments in three-dimensional space. Optionally, the method for obtaining sample line cloud data includes the following steps 1 to 4:

[0144] 1. Obtain a multi-view image set of a sample three-dimensional object corresponding to the sample three-dimensional model, where the multi-view image set includes multiple sample images obtained by photographing the sample three-dimensional object from multiple viewpoints;

[0145] 2. Detect and obtain multiple sample two-dimensional line segments from a multi-view image set; illustratively, based on at least one of the image's color information, depth information, and texture information, multiple sample two-dimensional line segments are detected from multiple sample images, where the sample two-dimensional line segments refer to line segments in a two-dimensional space.

[0146] 3. According to the internal and external parameters of the multi-view image set, multiple sample two-dimensional line segments are converted into multiple sample three-dimensional line segments.

[0147] 4. Determine multiple sample three-dimensional line segments as line cloud data.

[0148] Sample connection point set: In this application, the sample connection point set includes all the connection points for constructing a sample three-dimensional model. The sample connection points are the contour points of the sample three-dimensional model. All the contour points of the sample three-dimensional model constitute the point frame structure of the sample three-dimensional model.

[0149] For details of step 710 , please refer to the relevant introduction of the three-dimensional model, connection point set, and line cloud data in step 210 above.

[0150] Step 720: construct a target sample feature vector based on a sample connection point pair constructed from any two sample connection points in the sample connection point set, and the distance between the sample connection points in the sample connection point pair and the sample three-dimensional line segment in the sample line cloud data;

[0151] 1. For the nth connection point in the sample connection point set, determine L sample three-dimensional line segments from the sample line cloud data whose distance from the nth connection point is less than a distance threshold. The nth connection point is any connection point in the sample connection point set.

[0152] 2. According to the N connection points of the sample connection point set, the L sample three-dimensional line segments whose distance from the nth connection point is less than the distance threshold, and the line segment features of the lth sample three-dimensional line segment in the L sample three-dimensional line segments, a first sample feature vector represented as N×L×K is obtained.

[0153] 3. For the nth connection point, pair the connection points in the sample connection point set except the nth connection point with the nth connection point.

[0154] 4. Transform the first sample feature vector represented as N×L×K into The second sample feature vector of .

[0155] For details of step 720 , please refer to the description of constructing the target feature vector in step 220 .

[0156] Step 730, inputting the target sample feature vector into the prediction network;

[0157] In one embodiment, the prediction network is composed of at least one classification layer and an output layer stacked in sequence. The classification layer is composed of a fully connected layer, a batch normalization layer, a nonlinear layer, and a dropout layer stacked in sequence; the output layer includes a fully connected layer; wherein the number of output channels of the output layer is C.

[0158] Exemplarily, the prediction network is composed of three prediction layers and an output layer stacked in sequence. The prediction network includes the first prediction layer, the second prediction layer, the third prediction layer, and the output layer. The first prediction layer, the second prediction layer, and the third prediction layer all include a fully connected layer, a batch normalization layer, a nonlinear layer, and a dropout layer.

[0159] Schematically, the feature vector input to the prediction network 570 is represented as The feature vector output by the prediction network 570 is represented as Then the number of output channels of the first prediction layer is 128, the number of output channels of the second prediction layer is 64, the number of output channels of the third prediction layer is 32, and the number of output channels of the output layer 574 is 4.

[0160] Optionally, the second sample feature vector obtained in step 720 is input into the prediction network.

[0161] Step 740: predicting a first connection relationship corresponding to the sample connection point set through a prediction network;

[0162] The first connection relationship includes a connection relationship between any two sample connection points in the predicted sample connection point set on the sample three-dimensional model.

[0163] Optionally, the first connection relationship includes a Class C connection relationship. Optionally, the Class C connection relationship includes: the two connection points in the connection point pair are the same connection point; the two connection points in the connection point pair are directly connected; the two connection points in the connection point pair are connected through p connection points, where the value of p is any one from 1 to C-3, and C is a positive integer greater than 3; the two connection points in the connection point pair are connected through at least C-2 connection points.

[0164] For the convenience of introduction, the following example is given using a C-type connection relationship including: two connection points in a connection point pair are connected or not connected. Similarly, the training method when C is greater than 2 can be obtained.

[0165] Schematically, in a training process, the sample connection point set includes sample connection point 1, sample connection point 2, sample connection point 3, sample connection point 4, and sample connection point 5, and the predicted first connection relationships include: (1 and 2 are connected), (1 and 3 are not connected), (1 and 4 are not connected), (1 and 5 are connected), (2 and 3 are connected), (2 and 4 are connected), (2 and 5 are connected), (3 and 4 are connected), (3 and 5 are not connected), and (4 and 5 are connected).

[0166] Step 750: training a prediction network based on the second connection relationship and the first connection relationship of the sample three-dimensional model.

[0167] The second connection relationship includes the connection relationship between any two connection points in the sample connection point set obtained by marking.

[0168] Schematically, the second connection relationship includes (1 and 2 are connected), (1 and 3 are connected), (1 and 4 are connected), (1 and 5 are connected), (2 and 3 are connected), (2 and 4 are not connected), (2 and 5 are connected), (3 and 4 are connected), (3 and 5 are not connected), and (4 and 5 are connected).

[0169] By comparing the first connection relationship with the second connection relationship, it can be seen that the incorrect ones in the first connection relationship are "1 and 3 are not connected", "1 and 4 are not connected", and "2 and 4 are connected", while the correct ones in the first connection relationship are "1 and 2 are connected", "1 and 5 are connected", "2 and 3 are connected", "2 and 5 are connected", "3 and 4 are connected", "3 and 5 are not connected", and "4 and 5 are connected". Based on the incorrect and correct results, the parameters of the prediction network can be updated and the prediction network can be retrained.

[0170] In one embodiment, the second connection relationship is obtained by annotation based on the connection relationship of a real three-dimensional object. For example, in a real three-dimensional object, a contour point with coordinates (0, 0, 1.05) is connected to a contour point with coordinates (1, 1, 1.1). In the sample connection point set, there are a sample connection point with coordinates (0, 0, 1) and a sample connection point with coordinates (1, 1, 1). Because the distance between the point (0, 0, 1) and the point (0, 0, 1.05) is less than a threshold, and the distance between the point (1, 1, 1) and the point (1, 1, 1.1) is less than a threshold, then the point (0, 0, 1) and the point (1, 1, 1.1) are considered to be connected.

[0171] For example, in a real three-dimensional object, the contour point with coordinates (3, 3, 3.05) is not connected to the contour point with coordinates (5, 5, 5.1). In the sample connection point set, there are sample connection points with coordinates (3, 3, 3) and sample connection points with coordinates (5, 5, 5). Based on the fact that the distance between point (3, 3, 3) and point (3, 3, 3.05) is less than the threshold, and the distance between point (5, 5, 5) and point (5, 5, 5.1) is less than the threshold, it is considered that point (3, 3, 3) and point (5, 5, 5) are not connected.

[0172] In summary, a training method for a connection relationship prediction network is provided, which ensures that the prediction network can be used to predict connection relationships.

[0173] In step 720, after the second sample feature vector is constructed, at least one of the following steps is further performed:

[0174] (1) Input the second sample feature vector into the multilayer perceptron, and get the representation as The third sample eigenvector of , the value of K' is greater than K, and K' is a positive integer;

[0175] (2) Input the second sample feature vector into the first Transformer network, and the result is represented as The fourth sample feature vector of ; wherein the first Transformer network is used to add a self-attention mechanism in the number dimension of three-dimensional line segments.

[0176] (3) Input the second sample feature vector into the pooling network, and obtain the representation as The fifth sample eigenvector of ;

[0177] Optionally, a pooling network is used to perform average pooling in the dimension of the number of 3D line segments. The output of the pooling network is expressed as The fifth sample eigenvector of .

[0178] (4) Input the second sample feature vector into the second Transformer network, and obtain the representation: The sixth sample feature vector of ; wherein the second Transformer network is used to add a self-attention mechanism in the dimension of the number of connection point pairs.

[0179] In an optional embodiment, the connection relationship determination method provided in this application can be used for 3D reconstruction of real-world 3D objects. For example, the connection relationship determination method provided in this application can be used for 3D reconstruction in urban planning, surveying and mapping systems, high-precision maps, virtual simulation, cultural relics protection, VR (Virtual Reality) / AR (Augmented Reality) navigation and other fields.

[0180] Taking the field of urban planning as an example, a multi-perspective image set of real-world 3D buildings is first acquired. The multi-perspective image set includes multiple images of the 3D buildings captured from multiple perspectives. For example, a ground-based camera captures the 3D buildings from an upward perspective, a drone in the air captures the 3D buildings from a level perspective, and a drone in the air captures the 3D buildings from a downward perspective. After acquiring the multi-perspective image set, multiple 2D line segments are detected from the multi-perspective image set. These 2D line segments may include the outline segments of the 3D buildings or noise segments outside the 3D buildings. The multiple 2D line segments are converted into multiple 3D line segments. The multiple 3D line segments are then determined as line cloud data for the 3D buildings.

[0181] After obtaining the line cloud data of the 3D building, the connection point set of the constructed virtual 3D building will also be obtained. Optionally, the connection points of the virtual 3D building can be predicted by a connection point prediction network, or can be points selected by the developer based on the 3D building.

[0182] Any two connection points in the acquired connection point set are constructed into a connection point pair. A target feature vector is constructed based on the connection point pair corresponding to the connection point set and the distance between the connection points in the connection point pair and the 3D line segments in the line cloud data. The target feature vector is input into the connection relationship prediction network; the prediction network predicts the connection relationship between any two connection points in the connection point set in the virtual 3D building.

[0183] Taking the virtual simulation of a real human body as an example, a multi-view image set of a real-world human body is first acquired. The multi-view image set includes multiple images of the human body captured from multiple perspectives; for example, the human body may be photographed from the front, from the side, from the top of the head, and so on. After acquiring the multi-view image set, multiple two-dimensional line segments are detected from the multi-view image set. These two-dimensional line segments may include the human body's contour segments or external noise segments. For example, if a person is photographed standing on steps, the detected two-dimensional line segments are likely to include the edge lines of the steps. The multiple two-dimensional line segments are converted into multiple three-dimensional line segments. The multiple three-dimensional line segments are determined as line cloud data of the human body.

[0184] After obtaining the line cloud data of the human body, the connection point set of the constructed virtual object will also be obtained. Optionally, the connection points of the virtual object can be predicted by the connection point prediction network, or they can be points selected by the developer based on the human body.

[0185] Any two connection points in the acquired connection point set are constructed into a connection point pair. A target feature vector is constructed based on the connection point pair corresponding to the connection point set and the distance between the connection points in the connection point pair and the 3D line segments in the line cloud data. The target feature vector is input into the connection relationship prediction network; the prediction network predicts the connection relationship between any two connection points in the connection point set in the virtual object.

[0186] Figure 8 The following is a block diagram of a device for determining a connection relationship provided by an exemplary embodiment of the present application. The device includes:

[0187] An acquisition module 801 is configured to acquire line cloud data and a connection point set for reconstructing a 3D model. The line cloud data includes multiple 3D line segments associated with the 3D model. The connection points in the connection point set are contour points of the 3D model. The 3D line segments are line segments in 3D space.

[0188] A construction module 802 is configured to construct a target feature vector based on a connection point pair constructed from any two connection points in the connection point set and the distance between the connection points in the connection point pair and the three-dimensional line segment in the line cloud data;

[0189] An input module 803 is used to input the target feature vector into the prediction network of the connection relationship;

[0190] The prediction module 804 is used to predict the connection relationship between any two connection points in the connection point set in the three-dimensional model through a prediction network.

[0191] In an optional embodiment, the construction module 802 is further used to determine, for the nth connection point in the connection point set, L three-dimensional line segments whose distance from the nth connection point is less than a distance threshold from the line cloud data; the nth connection point is any connection point in the connection point set; based on the N connection points in the connection point set, the L three-dimensional line segments whose distance from the nth connection point is less than the distance threshold, and the line segment features of the lth three-dimensional line segment in the L three-dimensional line segments, obtain a first feature vector represented as N×L×K; the lth three-dimensional line segment is any one of the L three-dimensional line segments, and K is the number of features of the lth three-dimensional line segment; for the nth connection point, pair the connection points in the connection point set except the nth connection point with the nth connection point; transform the first feature vector represented as N×L×K into a feature vector represented as The second eigenvector of The number of tie point pairs obtained by pairing a set of tie points.

[0192] In an optional embodiment, the construction module 802 is further configured to input the second feature vector into a multilayer perceptron to obtain The third eigenvector of , K' has a value greater than K.

[0193] In an optional embodiment, the construction module 802 is further configured to input the second feature vector into the first transformer network to obtain The fifth eigenvector of ; wherein the first Transformer network is used to add a self-attention mechanism in the dimension of the number of three-dimensional line segments.

[0194] In an optional embodiment, the construction module 802 is further configured to input the second feature vector into the pooling network to obtain The fourth eigenvector of .

[0195] In an optional embodiment, the construction module 802 is further configured to input the second feature vector into the second Transformer network to obtain The sixth eigenvector of , where the second Transformer network is used to add a self-attention mechanism in the dimension of the number of connection point pairs.

[0196] In an optional embodiment, the prediction module 804 is further configured to obtain, through a prediction network, The seventh eigenvector of The probability distribution of connection point pairs on C types of connection relationships.

[0197] In an optional embodiment, the C-type connection relationship includes:

[0198] The two connection points in a connection point pair belong to the same connection point;

[0199] The two connection points in a connection point pair are directly connected;

[0200] The two connection points in a connection point pair are connected by p connection points, where the value of p is any one from 1 to C-3;

[0201] The two attachment points in a pair of attachment points are connected by at least C-2 attachment points.

[0202] In an optional embodiment, the prediction network is composed of at least one classification layer and an output layer stacked in sequence; the classification layer is composed of a fully connected layer, a batch normalization (BN) layer, a nonlinear layer, and a dropout layer stacked in sequence; the output layer includes a fully connected layer; wherein the number of output channels of the output layer is C.

[0203] In an optional embodiment, the acquisition module 801 is also used to obtain a multi-perspective image set of a three-dimensional object corresponding to the three-dimensional model, the multi-perspective image set including multiple images obtained by shooting the three-dimensional object from multiple perspectives; detecting multiple two-dimensional line segments from the multi-perspective image set; converting the multiple two-dimensional line segments into multiple three-dimensional line segments based on the internal and external parameters of the multi-perspective image set; and determining the multiple three-dimensional line segments as line cloud data.

[0204] In summary, based on the connection point pairs constructed from any two connection points in the connection point set, and the distances between the connection points in the connection point pairs and the three-dimensional line segments in the line cloud data, a target feature vector for predicting connection relationships is constructed. This provides a method for predicting connection relationships based on line cloud data and connection points, overcoming the problems in related technologies of difficulty in obtaining point cloud data and poor generalization of the connection point estimation network.

[0205] Figure 9 A training device for a connection relationship prediction network provided by an exemplary embodiment of the present application is shown, the device comprising:

[0206] An acquisition module 901 is configured to acquire sample line cloud data and a sample connection point set for reconstructing a sample 3D model. The sample line cloud data includes a plurality of sample 3D line segments associated with the sample 3D model. The sample connection points in the sample connection point set are contour points of the sample 3D model. The sample 3D line segments are line segments in 3D space.

[0207] A construction module 902 is configured to construct a target sample feature vector based on a sample connection point pair constructed from any two sample connection points in the sample connection point set, and the distance between the sample connection points in the sample connection point pair and the sample three-dimensional line segment in the sample line cloud data;

[0208] Input module 903, used to input the target sample feature vector into the prediction network;

[0209] A prediction module 904 is configured to predict, through a prediction network, a first connection relationship corresponding to the sample connection point set, where the first connection relationship includes a connection relationship between any two sample connection points in the predicted sample connection point set on the sample three-dimensional model;

[0210] The training module 905 is used to train the prediction network according to the second connection relationship and the first connection relationship of the sample connection point set, where the second connection relationship includes the connection relationship between any two sample connection points in the sample connection point set obtained by labeling.

[0211] In an optional embodiment, the construction module 902 is further configured to input the second sample feature vector into a multilayer perceptron to obtain The third sample eigenvector of , the value of K' is greater than K, and K' is a positive integer.

[0212] In an optional embodiment, the construction module 902 is further configured to input the second sample feature vector into the first transformer network to obtain The fourth sample feature vector of the first Transformer network is used to add a self-attention mechanism in the quantity dimension of the sample three-dimensional line segments.

[0213] In an optional embodiment, the construction module 902 is further configured to input the second sample feature vector into the pooling network to obtain The fifth sample eigenvector of .

[0214] In an optional embodiment, the construction module 902 is further configured to input the second sample feature vector into the second Transformer network to obtain The sixth sample feature vector of ; wherein the second Transformer network is used to add a self-attention mechanism in the dimension of the number of connection point pairs.

[0215] In summary, a training device for a connection relationship prediction network is provided, which ensures that the prediction network can be used to predict connection relationships.

[0216] Figure 10 1 is a schematic diagram illustrating the structure of a computer device according to an exemplary embodiment. The computer device 1000 includes a central processing unit (CPU) 1001, a system memory 1004 including a random access memory (RAM) 1002 and a read-only memory (ROM) 1003, and a system bus 1005 connecting the system memory 1004 and the CPU 1001. The computer device 1000 also includes a basic input / output system (I / O system) 1006 for facilitating information transmission between various components within the computer device, and a mass storage device 1007 for storing an operating system 1013, application programs 1014, and other program modules 1015.

[0217] The basic input / output system 1006 includes a display 1008 for displaying information and an input device 1009, such as a mouse or keyboard, for user input. Both the display 1008 and the input device 1009 are connected to the central processing unit 1001 via an input / output controller 1010 connected to the system bus 1005. The basic input / output system 1006 may also include an input / output controller 1010 for receiving and processing input from a variety of other devices, such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 1010 also provides output to a display screen, printer, or other types of output devices.

[0218] The mass storage device 1007 is connected to the central processing unit 1001 via a mass storage controller (not shown) connected to the system bus 1005. The mass storage device 1007 and its associated computer-readable medium provide non-volatile storage for the computer device 1000. In other words, the mass storage device 1007 may include a computer-readable medium (not shown) such as a hard disk or a Compact Disc Read-Only Memory (CD-ROM) drive.

[0219] Without loss of generality, the computer device readable medium may include computer device storage media and communication media. Computer device storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer device readable instructions, data structures, program modules or other data. Computer device storage media include RAM, ROM, Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), CD-ROM, Digital Video Disc (DVD) or other optical storage, tape cassettes, magnetic tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer device storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1004 and mass storage device 1007 can be collectively referred to as memory.

[0220] According to various embodiments of the present disclosure, the computer device 1000 may also be connected to a remote computer device on a network such as the Internet for operation. That is, the computer device 1000 may be connected to the network 1011 via the network interface unit 1012 connected to the system bus 1005, or the network interface unit 1012 may be used to connect to other types of networks or remote computer device systems (not shown).

[0221] The memory further includes one or more programs, which are stored in the memory. The central processing unit 1001 implements all or part of the steps of the above-mentioned method for determining the connection relationship by executing the one or more programs.

[0222] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, the at least one program, the code set or instruction set is loaded and executed by a processor to implement the method for determining the connection relationship provided in the above method embodiment, or the training method for the connection relationship prediction network.

[0223] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for determining a connection relationship or the method for training a network for predicting a connection relationship provided in the above-mentioned method embodiments.

[0224] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0225] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0226] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for determining a connection relationship, characterized in that: The method comprises: Acquire a multi-view image set of a three-dimensional object corresponding to the three-dimensional model, wherein the multi-view image set includes a plurality of images of the three-dimensional object obtained by photographing the three-dimensional object from multiple viewpoints; Detecting and obtaining a plurality of two-dimensional line segments from the multi-view image set; converting the plurality of two-dimensional line segments into a plurality of three-dimensional line segments according to the internal and external parameters of the multi-view image set; and determining the plurality of three-dimensional line segments as line cloud data for reconstructing the three-dimensional model; Acquire a connection point set for reconstructing the three-dimensional model, where the connection points in the connection point set are contour points of the three-dimensional model, and the three-dimensional line segments are line segments in three-dimensional space; constructing a target feature vector based on a connection point pair constructed from any two connection points in the connection point set and a distance between a connection point in the connection point pair and a three-dimensional line segment in the line cloud data; Inputting the target feature vector into the prediction network of the connection relationship; The connection relationship between any two connection points in the connection point set in the three-dimensional model is obtained through the prediction network.

2. The method according to claim 1, characterized in that The target feature vector is constructed based on a connection point pair constructed from any two connection points in the connection point set, and the distance between the connection points in the connection point pair and the three-dimensional line segment in the line cloud data, including: For an nth connection point in the connection point set, determining L three-dimensional line segments whose distances from the nth connection point are less than a distance threshold from the line cloud data; the nth connection point is any connection point in the connection point set, and n and L are positive integers; A first feature vector represented as N×L×K is obtained based on the N connection points of the connection point set, L three-dimensional line segments whose distances from the nth connection point are less than a distance threshold, and the segment features of the lth three-dimensional line segment among the L three-dimensional line segments; the lth three-dimensional line segment is any one of the L three-dimensional line segments, K is the number of features of the lth three-dimensional line segment, and L, l, and K are all positive integers; For the nth connection point, pairing connection points in the connection point set other than the nth connection point with the nth connection point; The first eigenvector represented as N×L×K is transformed into a first eigenvector represented as The second eigenvector of The number of join point pairs obtained by pairing the join point set.

3. The method according to claim 2, characterized in that The method further comprises: The second feature vector is input into the multilayer perceptron, which is represented as The third eigenvector of , the value of K' is greater than K, and K' is a positive integer; The third eigenvector is determined as the target eigenvector.

4. The method according to claim 2, characterized in that The method further comprises: The second feature vector is input into the first Transformer network, and the representation is obtained as The fourth eigenvector of ; Determining the fourth eigenvector as the target eigenvector; The first Transformer network is used to add a self-attention mechanism in the quantity dimension of the three-dimensional line segments.

5. The method according to claim 2, characterized in that The method further comprises: The second feature vector is input into the pooling network, and is expressed as The fifth eigenvector of ; The fifth eigenvector is determined as the target eigenvector.

6. The method according to claim 2, characterized in that The method further comprises: The second feature vector is input into the second Transformer network, and the representation is obtained as The sixth eigenvector of ; determining the sixth eigenvector as the target eigenvector; The second Transformer network is used to add a self-attention mechanism in the dimension of the number of connection point pairs.

7. The method according to claim 2, characterized in that The step of obtaining, through the prediction network, a connection relationship between any two connection points in the connection point set in the three-dimensional model includes: Through the prediction network, we can get The seventh eigenvector of The probability distribution of connection point pairs on C types of connection relationships, where C is a positive integer.

8. The method according to claim 7, characterized in that The Class C connection relationship includes: The two connection points in the connection point pair belong to the same connection point; The two connection points in the connection point pair are directly connected; The two connection points in the connection point pair are connected by p connection points, where the value of p is any one from 1 to C-3, and C is a positive integer greater than 3; The two attachment points in the pair of attachment points are connected by at least C-2 attachment points.

9. The method according to claim 7, characterized in that The prediction network is composed of at least one classification layer and an output layer stacked in sequence; The classification layer is composed of a fully connected layer, a batch normalization (BN) layer, a nonlinear layer, and a dropout layer stacked in sequence; The output layer includes a fully connected layer; The number of output channels of the output layer is C.

10. A method for training a connection relationship prediction network, characterized in that: The method comprises: Acquire a multi-view image set of a sample three-dimensional object corresponding to the sample three-dimensional model, wherein the multi-view image set includes a plurality of sample images obtained by photographing the sample three-dimensional object from multiple viewpoints; Detecting and obtaining a plurality of sample two-dimensional line segments from the multi-view image set; converting the plurality of sample two-dimensional line segments into a plurality of sample three-dimensional line segments according to the internal and external parameters of the multi-view image set; and determining the plurality of sample three-dimensional line segments as sample line cloud data for reconstructing the sample three-dimensional model; Acquire a sample connection point set for reconstructing the sample three-dimensional model, where the sample connection points in the sample connection point set are contour points of the sample three-dimensional model, and the sample three-dimensional line segments refer to line segments in three-dimensional space; Constructing a target sample feature vector based on a sample connection point pair constructed from any two sample connection points in the sample connection point set, and a distance between a sample connection point in the sample connection point pair and a sample three-dimensional line segment in the sample line cloud data; Inputting the target sample feature vector into the prediction network; Predicting a first connection relationship corresponding to the sample connection point set through the prediction network, where the first connection relationship includes a predicted connection relationship between any two sample connection points in the sample connection point set on the sample three-dimensional model; The prediction network is trained according to a second connection relationship of the sample connection point set and the first connection relationship, where the second connection relationship includes a connection relationship between any two sample connection points in the sample connection point set obtained by labeling.

11. A device for determining a connection relationship, characterized in that: The device comprises: an acquisition module configured to acquire a multi-view image set of a three-dimensional object corresponding to a three-dimensional model, the multi-view image set comprising a plurality of images of the three-dimensional object captured from multiple viewpoints; detect a plurality of two-dimensional line segments from the multi-view image set; convert the plurality of two-dimensional line segments into a plurality of three-dimensional line segments based on internal and external parameters of the multi-view image set; and determine the plurality of three-dimensional line segments as line cloud data for reconstructing the three-dimensional model; The acquisition module is configured to acquire a set of connection points for reconstructing the three-dimensional model, wherein the connection points in the set of connection points are contour points of the three-dimensional model, and the three-dimensional line segments are line segments in the three-dimensional space; a construction module, configured to construct a target feature vector based on a connection point pair constructed from any two connection points in the connection point set, and a distance between a connection point in the connection point pair and a three-dimensional line segment in the line cloud data; An input module, configured to input the target feature vector into the prediction network of the connection relationship; The prediction module is used to obtain the connection relationship between any two connection points in the connection point set in the three-dimensional model through the prediction network.

12. A training device for a connection relationship prediction network, characterized in that: The device comprises: an acquisition module configured to acquire a multi-view image set of a sample three-dimensional object corresponding to a sample three-dimensional model, the multi-view image set comprising a plurality of sample images obtained by photographing the sample three-dimensional object from multiple viewpoints; detect a plurality of sample two-dimensional line segments from the multi-view image set; convert the plurality of sample two-dimensional line segments into a plurality of sample three-dimensional line segments based on internal and external parameters of the multi-view image set; and determine the plurality of sample three-dimensional line segments as sample line cloud data for reconstructing the sample three-dimensional model; The acquisition module is configured to acquire a set of sample connection points for reconstructing the sample three-dimensional model, wherein the sample connection points in the set of sample connection points are contour points of the sample three-dimensional model, and the sample three-dimensional line segments are line segments in three-dimensional space; a construction module, configured to construct a target sample feature vector based on a sample connection point pair constructed from any two sample connection points in the sample connection point set, and a distance between the sample connection points in the sample connection point pair and the sample three-dimensional line segment in the sample line cloud data; An input module, configured to input the target sample feature vector into the prediction network; a prediction module, configured to predict, through the prediction network, a first connection relationship corresponding to the set of sample connection points, the first connection relationship comprising a predicted connection relationship between any two sample connection points in the set of sample connection points on the sample three-dimensional model; A training module is used to train the prediction network based on a second connection relationship of the sample connection point set and the first connection relationship, where the second connection relationship includes a connection relationship between any two sample connection points in the sample connection point set obtained by labeling.

13. A computer device, characterized in that: The computer device includes: a processor and a memory, the memory storing a computer program, and the computer program is loaded and executed by the processor to implement the method for determining the connection relationship according to any one of claims 1 to 9, or the method for training a network for predicting the connection relationship according to claim 10.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the method for determining a connection relationship according to any one of claims 1 to 9, or the method for training a connection relationship prediction network according to claim 10.

15. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. The processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes to implement the method for determining the connection relationship as described in any one of claims 1 to 9, or the training method for the prediction network of the connection relationship as described in claim 10.

Images