A scene semantic segmentation method, system, terminal device and readable storage medium

Abstract

The present invention discloses a scene semantic segmentation method, system, terminal device and readable storage medium. The input of the present invention is a three-dimensional indoor scene model, and the output is a semantic segmentation result of a scene graph. First, it is necessary to extract the trajectory features of the movement of characters in the three-dimensional scene. In this step, it is necessary to evenly collect the interaction points between people and the scene in the walkable area of ​​the scene, and then use the navigation grid algorithm to simulate and calculate the path distance of the two interaction points. After obtaining the path distance of the two interaction points, the connectivity of each interaction point is calculated. Between the interaction points and the interaction points, an interaction point scene graph can be constructed by a weight value based on connectivity. Finally, a normalized cut algorithm is used to segment the weighted scene graph, thereby completing the semantic segmentation of the three-dimensional scene.

Description

Technical Field

[0001] The present invention belongs to the field of three-dimensional scene understanding, and relates to a scene semantic segmentation method, system, terminal equipment and readable storage medium. Background Art

[0002] Indoor 3D scenes are an important part of the 3D virtual world. They can not only be used to present the environment in virtual reality and film and television animation, but also are the main body of interaction in applications with interactive requirements such as robots, games, and education. Faced with a large amount of 3D scene data, how to represent and analyze it to facilitate data understanding is an important research direction.

[0003] Early scene representation and analysis methods were mainly based on the analysis of scene geometry. The most typical idea is to represent the objects in the scene and the spatial relationships between different objects by establishing a scene graph. According to the tightness of the spatial relationship, it can be divided into simple spatial relationship representation and complex spatial relationship representation.

[0004] In order to analyze the overall or local functions of a scene, researchers introduce people into the scene and understand the scene by jointly modeling and analyzing the information about people and the scene. Related research on the carrying function of the environment has laid a theoretical foundation for this type of scene analysis method. Subsequent research includes adding reasonable human interactions to two-dimensional scenes and analyzing RGB-D scenes.

[0005] In the above work, the interaction features are mainly represented by static human posture and surrounding environment geometry. This method that only considers static interaction does not encode the motion features in the scene, which makes it difficult for the algorithm to analyze some unreasonable layouts of the scene. For example, a part of the scene is closed and inaccessible; or a necessary path in the center of the scene is too narrow. Summary of the invention

[0006] The purpose of the present invention is to solve the problems in the prior art and provide a scene semantic segmentation method, system, terminal device and readable storage medium based on connectivity and normalized segmentation. Inspired by the research related to the analysis of human motion in the scene, the present invention considers the dynamic interaction characteristics between people and scenes in scene analysis. The motion trajectory of a person contains rich interaction information between the scene and the person, which plays a very important role in the semantic understanding of the scene.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides a scene semantic segmentation method, comprising the following steps:

[0009] Triangulate the input 3D model scene to generate several navigation meshes;

[0010] The union of several navigation networks constitutes a walkable area, and points are evenly sampled in the walkable area to obtain the coordinates of the sample points. All sample points constitute the interactive point set of the scene;

[0011] The A-Star algorithm and the corner point algorithm are used to calculate the simulated path in the navigation grid, and the shortest paths between the sample points in all the interaction points are calculated respectively;

[0012] The connectivity value of each sample point is obtained based on the shortest path calculation;

[0013] According to the connectivity value of each sample point, the weight of each edge is calculated to construct the similarity matrix of the undirected graph;

[0014] Scene semantic segmentation on undirected graphs using normalized segmentation.

[0015] In a second aspect, the present invention provides a scene semantic segmentation system, comprising:

[0016] The navigation network production module is used to triangulate the input 3D model scene and generate several navigation meshes;

[0017] The point sampling module is used to form a walkable area by combining the union of several navigation networks, uniformly sampling points in the walkable area to obtain the coordinates of the sample points, and all the sample points constitute the interactive point set of the scene;

[0018] The shortest path calculation module is used to calculate the simulation path in the navigation grid using the A-Star algorithm and the corner point algorithm, and to calculate the shortest path between each pair of sample points in all interaction point sets;

[0019] Connectivity value calculation module, used to calculate the connectivity value of each sample point according to the shortest path;

[0020] The similarity matrix construction module is used to calculate the weight of each edge according to the connectivity value of each sample point, so as to construct the similarity matrix of the undirected graph;

[0021] Semantic Segmentation Module, which is used to perform scene semantic segmentation on undirected graphs using normalized segmentation.

[0022] In a third aspect, the present invention provides a mobile terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0023] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The present invention simulates the character's motion path through a navigation grid algorithm, calculates the path length between the interaction points in the scene, and then calculates the connectivity of each interaction point, simulates the trajectory and connectivity index, and contains rich interaction information between the scene and the character. By using the simulated trajectory and connectivity, the present invention can effectively extract the interaction information between people and the environment in a complex three-dimensional scene model, thereby simplifying the original data and extracting key information useful for scene segmentation. By using sample points, connectivity and path distance, the scene is simply and efficiently represented as a scene graph, and finally a graph segmentation algorithm based on normalized segmentation is used to complete the semantic segmentation of the scene graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 Flowchart of the scene semantic segmentation method of the present invention.

[0028] Figure 2 Schematic diagram of the scene semantic segmentation system of the present invention.

[0029] Figure 3 is a flow chart of the method of the present invention.

[0030] Figure 4 It is a schematic diagram of the navigation grid and walkable area at different heights generated by the present invention.

[0031] Figure 5 It is a schematic diagram of the sample collection point of the present invention.

[0032] Figure 6 It is a schematic diagram of the shortest path calculated by the present invention.

[0033] Figure 7 It is a visualization diagram of the connectivity calculated by the present invention.

[0034] Figure 8 It is the semantic segmentation result of the present invention. Among them, (a) is the ncut connectivity segmentation map, (b) is the segmentation annotation ground truth map, (c) is the scene rendering map, (d) is the ncut connectivity segmentation map, (e) is the segmentation annotation groundtruth map, and (f) is the scene rendering map. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0036] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0038] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0039] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0040] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0041] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0042] See also Figure 1 The embodiment of the present invention discloses a scene semantic segmentation method, comprising the following steps:

[0043] S1 triangulates the input 3D model scene and generates several navigation meshes; the details are as follows:

[0044] The walkable area at a certain height of the model data is triangulated to obtain the navigation mesh at that height; the external form of the navigation mesh is a series of mutually adjacent convex polygons. There are no obstacles inside the convex polygons, and the character can move inside a convex polygon.

[0045] S2 forms a walkable area by combining the union of several navigation networks, uniformly samples points in the walkable area, obtains the coordinates of the sample points, and all the sample points constitute the interactive point set of the scene;

[0046] The top view of the walkable area of ​​the scene is evenly divided into N×N squares, with the side length of each square being 0.36m. The center coordinates of the square are taken as the coordinates of the collected sample points, and all sample points constitute the set of all interaction points of the scene.

[0047] S3 uses the A-Star algorithm and the corner point algorithm to calculate the simulated path in the navigation grid, and calculates the shortest path between each pair of sample points in all interaction point sets;

[0048] (1) Use the A-Star algorithm to traverse the grid and find a polygon sequence from the starting point to the end point by minimizing the cost function. The polygon sequence constitutes a channel;

[0049] The cost function f(n) is calculated as shown in formula (1):

[0050] f(n)=g(n)+h(n) (1)

[0051] Where: g(n) is the actual path length from the initial point to the current position, h(n) is the estimated distance of the expected optimal path from the current point to the end point; g(n) is set to the sum of the lengths of the line segments between the starting point and the geometric center of the triangle passed through, and h(n) is set to the Euclidean distance between the current point and the end point; by calculating the f value of the neighboring nodes of the current node, the node with the smallest f value among the neighboring nodes is selected as the next node of the current node, so as to determine a set of sequences from the starting node to the end node;

[0052] After obtaining a set of node sequences from the starting point to the end point through the A-Star algorithm, since each node represents a triangle, a set of triangle sequences is obtained. By expanding this set of triangle sequences, a channel from the starting point to the end point is obtained.

[0053] (2) Use the corner point algorithm to determine the shortest polyline path in a channel; the input of the algorithm is a triangle sequence, and the output is a polyline path from the starting point to the end point. The steps are as follows:

[0054] 1) Take the starting point as the vertex and the endpoints of the common edge of the starting triangle and the next triangle as two vertices to construct a funnel-shaped area;

[0055] 2) Determine whether the next node is inside the initial funnel-shaped area; if the left side is inside, move the left side line inward; if the right side is inside, move the right side inward;

[0056] 3) If a left node is below the right boundary of the funnel area, the next corner node can be obtained; similarly, if the funnel area of ​​a right node is below the left boundary, the next corner node can be obtained;

[0057] 4) Repeat the above operation to obtain all corner nodes;

[0058] 5) Connecting all the inflection points in sequence will form a broken line path.

[0059] S4 obtains the connectivity value of each sample point based on the shortest path calculation;

[0060] The connectivity of each sample point is calculated according to the following formula:

[0061]

[0062] Where: N represents the number of sample points in the scene, path(i, j) represents the shortest path length from sample point i to sample point j, dis(i, j) represents the Euclidean distance from point i to point j; ε(j!=i) is an indicator function, when j=i, its value is 0, when j!=i, its value is 1.

[0063] S5 calculates the weight of each edge according to the connectivity value of each sample point, thereby constructing a similarity matrix of the undirected graph;

[0064] The sample points uniformly collected in the scene are used as the vertices of the constructed graph, and the weights of the edges between vertices are constructed using the difference in connectivity between the points. The similarity matrix W is constructed as shown in formula (3):

[0065]

[0066] Where: w ij represents the element value of the i-th row and j-th column of the similarity matrix W, P ij Represents the path length between point i and point j, r and σ X is the parameter, c i and cj Represent the connectivity of sample points i and j respectively.

[0067] S6 performs scene semantic segmentation on undirected graphs using normalized segmentation.

[0068] (1) Calculate the Laplacian matrix of the similarity matrix W. The calculation formula is L = DW, where D is the degree matrix of the similarity matrix W. That is, add up the elements of each column of the similarity matrix W and put them on the diagonal, and the remaining elements are 0;

[0069] (2) Calculate the first k eigenvectors u1, ..., u in the generalized eigenvalue problem Lu = λDu k ; with u1,…,u k Construct a matrix U∈R for column vectors n×k ;

[0070] (3) Let y i is the i-th row vector of the matrix U, where i = 1, ..., n; use the k-means algorithm to sort the point set (y i ) i＝1，…，n Clustering is performed to obtain clustering results. Different clustering categories represent different rooms in the scene, thus achieving segmentation of the three-dimensional scene.

[0071] like Figure 2 As shown, an embodiment of the present invention discloses a scene semantic segmentation system, including:

[0072] The navigation network production module is used to triangulate the input 3D model scene and generate several navigation meshes;

[0073] The point sampling module is used to form a walkable area by combining the union of several navigation networks, uniformly sampling points in the walkable area to obtain the coordinates of the sample points, and all the sample points constitute the interactive point set of the scene;

[0074] The shortest path calculation module is used to calculate the simulation path in the navigation grid using the A-Star algorithm and the corner point algorithm, and to calculate the shortest path between each pair of sample points in all interaction point sets;

[0075] Connectivity value calculation module, used to calculate the connectivity value of each sample point according to the shortest path;

[0076] The similarity matrix construction module is used to calculate the weight of each edge according to the connectivity value of each sample point, so as to construct the similarity matrix of the undirected graph;

[0077] Semantic Segmentation Module, which is used to perform scene semantic segmentation on undirected graphs using normalized segmentation.

[0078] Example:

[0079] Figure 3 It is the overall flow chart of the present invention. The present invention is a scene semantic segmentation method based on connectivity and normalized segmentation, whose input is a three-dimensional indoor scene model, and the output is a semantic segmentation result of the scene graph. First, it is necessary to evenly collect the interaction points between people and the scene in the walkable area of ​​the scene, and then use the navigation grid algorithm to simulate and calculate the path distance of the two interaction points, so as to extract the motion trajectory characteristics of the characters in the three-dimensional scene. After obtaining the path distance of the two interaction points, the connectivity of each interaction point is calculated. Between the interaction points and the interaction points, an interaction point scene graph can be constructed by a weight value based on connectivity. Finally, a normalized-cut algorithm is used to segment the weighted scene graph to complete the semantic segmentation of the three-dimensional scene.

[0080] Step 1: Process the input 3D model scene to generate a navigation mesh. The external form of the navigation mesh is a series of adjacent convex polygons. There are no obstacles inside these convex polygons, and the characters can move freely inside a polygon.

[0081] By triangulating the walkable area at a certain height of the model data, the navigation mesh at that height can be obtained. Therefore, different navigation meshes can be calculated at different heights, such as Figure 4 shown.

[0082] Step 2: The union of all navigation grids constitutes the walkable area. Sample points are evenly collected in the walkable area to obtain the coordinates of the sample points. All sample points constitute the interaction point set of the scene.

[0083] After the navigation grid is generated, it is necessary to collect points in the walkable area represented by the navigation grid. The present invention adopts a uniform sampling method, that is, the top view of the walkable area of ​​the scene is evenly divided into N×N small squares. Considering that the walking step length of an adult is about 0.4m, for the convenience of calculation, the side length of each square is taken as 0.36m, and then the center coordinates of the small square are taken as the coordinates of the collected sample points, such as Figure 5 shown.

[0084] Step 3: Use the A-Star algorithm and the corner point algorithm to calculate the simulated path in the navigation grid, and calculate the shortest path between all sample points.

[0085] The navigation grid pathfinding algorithm is a heuristic algorithm for pathfinding on a two-dimensional plane defined by a navigation grid. Its core is the A-Star algorithm and the corner point algorithm.

[0086] The first step of the navigation grid pathfinding algorithm is to use the A-Star algorithm to traverse the grid and find a sequence of polygons from the starting point to the end point by minimizing the cost function. These polygon sequences can form a wider channel.

[0087] The cost function f(n) is calculated as shown in formula (1):

[0088] f(n)=g(n)+h(n) (1)

[0089] Where: g(n) is the actual path length from the initial point to the current position, and h(n) is the estimated distance of the optimal path from the current point to the end point. g(n) is set to the sum of the lengths of the line segments between the starting point and the geometric center of the triangle passed through, while h(n) is set to the Euclidean distance between the current point and the end point. By calculating the f-value of the neighboring nodes of the current node, the node with the smallest f-value among the neighboring nodes is selected as the next node of the current node, so as to determine a set of sequences from the starting node to the end node.

[0090] After obtaining a set of node sequences from the starting point to the end point through the A-Star algorithm, since each node represents a triangle, a set of triangle sequences is obtained. By expanding this set of triangle sequences, a channel from the starting point to the end point is obtained. The second step of the navigation grid algorithm is to calculate the shortest path in the channel obtained in the first step.

[0091] The corner point algorithm can be used to determine the shortest polyline path in a channel. The input of the algorithm is a triangle sequence, and the output is a polyline path from the starting point to the end point. The steps are as follows:

[0092] 1) Take the starting point as the vertex and the endpoints of the common edge of the starting triangle and the next triangle as two vertices to construct a funnel-shaped area.

[0093] 2) Determine whether the next node is inside the initial funnel-shaped area. If the left side is inside, move the left side line inward; if the right side is inside, move the right side inward.

[0094] 3) If a left node is below the right boundary of the funnel area, the next corner node can be obtained. Similarly, if the funnel area of ​​a right node is below the left boundary, the next corner node can be obtained.

[0095] 4) Repeat the above operation to get all corner nodes

[0096] 5) Connecting all the inflection points in sequence is the broken line path

[0097] like Figure 6 As shown, the broken line in the figure is the shortest path calculated based on the corner point algorithm.

[0098] Step 4: Calculate the connectivity value of each sample point based on the calculated shortest path.

[0099] After obtaining the path, the connectivity of each interaction point is calculated according to formula (2).

[0100]

[0101] Where: N represents the number of sample points in the scene, path(i, j) represents the shortest path length from point i to point j, and dis(i, j) represents the Euclidean distance from point i to point j. ε(j!=i) is an indicator function, which is 0 when j=i and 1 when j!=i.

[0102] Through experiments, we found that the connectivity in the bathroom, the bedroom at the side, the narrow closed kitchen and other areas is relatively poor, while in the public areas such as the living room and dining room, the connectivity is usually better. It can be seen that connectivity can also quantitatively describe the privacy of the indoor layout. Some areas in the scene are relatively open and can easily reach other areas, while some areas are relatively closed and need to go through a certain detour to go to other places. Figure 6 As shown, the connectivity of the interaction points located in the open area in the center of the scene is good, while the connectivity of the interaction points located in the edge area in the upper right corner of the scene is poor.

[0103] Normalize the connectivity to [0-1], set 0 to standard red, 1 to standard blue, and follow the uniform gradient of the color band. The visualization results are as follows: Figure 7 As shown, we can see that the connectivity is poor in the small area at the end of the scene and the closed area in the scene, while the connectivity is good in the more open area in the center of the scene.

[0104] Step 5: Set a weighted undirected graph G = (V, E), set the weight of each edge, calculate the weight of each edge according to the connectivity value of each sample point, and thus construct the similarity matrix W of the undirected graph G.

[0105] The present invention innovatively uses sample points uniformly collected in the scene as vertices of the constructed graph, and uses the difference in connectivity between points to construct the weights of the edges between vertices. The construction method of the similarity matrix W is shown in formula (3):

[0106]

[0107] Where: w ij represents the element value of the i-th row and j-th column of the similarity matrix W, P ij Represents the path length between point i and point j, r and σ X is the parameter, ci and c j Represent the connectivity of sample points i and j respectively.

[0108] Step 6: Use normalized segmentation to perform scene semantic segmentation on the scene graph obtained in step 5, including:

[0109] (6a) Calculate the Laplacian matrix of the similarity matrix W, and the calculation formula is L=DW, where D is the degree matrix of W, that is, add up the elements of each column of W and put them on the diagonal, and the remaining elements are 0.

[0110] (6b) Compute the first k eigenvectors u1, ..., u in the generalized eigenvalue problem Lu = λDu. k . With u1,…,u k Construct a matrix U∈R for column vectors n×k .

[0111] (6c) Let y i is the i-th row vector of the matrix U, where i = 1, ..., n. Use the k-means algorithm to classify the point set (y i ) i＝1，…，n Clustering is performed to obtain clustering results. Different clustering categories represent different rooms in the scene, thus segmenting the three-dimensional scene. Figure 8 As shown, the left column is the scene segmentation map of the present invention, the middle column is the Ground-Truth, and the right column is the actual scene rendering map. The present invention can well find the narrow places in the scene (such as Figure 8 The three-dimensional scene is semantically segmented using the segmentation points.

[0112] Principle of the present invention:

[0113] The input of the present invention is a three-dimensional indoor scene model, and the output is the semantic segmentation result of the scene graph. First, it is necessary to extract the trajectory features of the character's movement from the three-dimensional scene. In this step, it is necessary to evenly collect the interaction points between people and the scene in the walkable area of ​​the scene, and then use the navigation grid algorithm to simulate and calculate the path distance of the two interaction points. After obtaining the path distance of the two interaction points, the connectivity of each interaction point can be calculated. Between the interaction points and the interaction points, an interaction point scene graph can be constructed by a weight value based on connectivity. Finally, the scene can be semantically segmented effectively by using a normalized-cut algorithm on the weighted scene graph.

[0114] An embodiment of the present invention provides a mobile terminal. The mobile terminal of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of each module / unit in the above-mentioned device embodiments are implemented.

[0115] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0116] The mobile terminal may be a computing device such as a desktop computer, a notebook, a palm computer, a cloud server, etc. The mobile terminal may include, but is not limited to, a processor and a memory.

[0117] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0118] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the mobile terminal by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0119] If the module / unit integrated in the mobile terminal is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0120] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A scene semantic segmentation method, characterized in that: The following steps are involved: Triangulate the input 3D model scene to generate several navigation meshes; The union of several navigation networks constitutes a walkable area, and points are evenly sampled in the walkable area to obtain the coordinates of the sample points. All sample points constitute the interactive point set of the scene; The A-Star algorithm and the corner point algorithm are used to calculate the simulated path in the navigation grid, and the shortest paths between the sample points in all the interaction points are calculated respectively; The connectivity value of each sample point is obtained based on the shortest path calculation; According to the connectivity value of each sample point, the weight of each edge is calculated to construct the similarity matrix of the undirected graph; Scene semantic segmentation on undirected graphs using normalized segmentation.

2. The scene semantic segmentation method according to claim 1, characterized in that: The triangulation process of the input three-dimensional model scene includes: The walkable area at a certain height of the model data is triangulated to obtain the navigation mesh at that height; the external form of the navigation mesh is a series of mutually adjacent convex polygons. There are no obstacles inside the convex polygons, and the character can move inside a convex polygon.

3. The scene semantic segmentation method according to claim 1, characterized in that: The uniform sampling of points in the walkable area includes: The top view of the walkable area of ​​the scene is evenly divided into N×N squares, with the side length of each square being 0.36m. The center coordinates of the square are taken as the coordinates of the collected sample points, and all sample points constitute the set of all interaction points of the scene.

4. The scene semantic segmentation method according to claim 1, characterized in that: The A-Star algorithm and the corner point algorithm are used to calculate the simulated path in the navigation grid, and the shortest paths between the sample points in all the interaction point sets are calculated respectively, including: (1) Use the A-Star algorithm to traverse the grid and find a polygon sequence from the starting point to the end point by minimizing the cost function. The polygon sequence constitutes a channel; The cost function f(n) is calculated as shown in formula (1): f(n)=g(n)+h(n) (I) Where: g(n) is the actual path length from the initial point to the current position, h(n) is the estimated distance of the expected optimal path from the current point to the end point; g(n) is set to the sum of the lengths of the line segments between the starting point and the geometric center of the triangle passed through, and h(n) is set to the Euclidean distance between the current point and the end point; by calculating the f value of the neighboring nodes of the current node, the node with the smallest f value among the neighboring nodes is selected as the next node of the current node, so as to determine a set of sequences from the starting node to the end node; After obtaining a set of node sequences from the starting point to the end point through the A-Star algorithm, since each node represents a triangle, a set of triangle sequences is obtained. By expanding this set of triangle sequences, a channel from the starting point to the end point is obtained. (2) Use the corner point algorithm to determine the shortest polyline path in a channel; the input of the algorithm is a triangle sequence, and the output is a polyline path from the starting point to the end point. The steps are as follows: 1) Take the starting point as the vertex, and the endpoints of the common edge of the starting triangle and the next triangle as two vertices to construct a funnel-shaped area; 2) Determine whether the next node is inside the initial funnel-shaped area; if the left side is inside, move the left side line inward; if the right side is inside, move the right side inward; 3) If a left node is below the right boundary of the funnel area, the next corner node can be obtained; similarly, if the funnel area of ​​a right node is below the left boundary, the next corner node can be obtained; 4) Repeat the above operation to obtain all corner nodes; 5) Connecting all the inflection points in sequence will form a broken line path.

5. The scene semantic segmentation method according to claim 1, characterized in that: The connectivity value of each sample point is obtained by calculating the shortest path, including: The connectivity of each sample point is calculated according to the following formula: Where: N represents the number of sample points in the scene, path(i, j) represents the shortest path length from sample point i to sample point j, dis(i, j) represents the Euclidean distance from point i to point j; ε(j!=i) is an indicator function, when j=i, its value is 0, when j!=i, its value is 1.

6. The scene semantic segmentation method according to claim 1, characterized in that: According to the connectivity value of each sample point, the weight of each edge is calculated to construct the similarity matrix W of the undirected graph G, including: The sample points uniformly collected in the scene are used as the vertices of the constructed graph, and the weights of the edges between vertices are constructed using the difference in connectivity between the points. The similarity matrix W is constructed as shown in formula (3): Where: w ij represents the element value of the i-th row and j-th column of the similarity matrix W, P ij represents the path length between point i and point j, r and σ X is the parameter, c i and c j Represent the connectivity of sample points i and j respectively.

7. The scene semantic segmentation method according to claim 1, characterized in that: The method of performing scene semantic segmentation on an undirected graph using normalized segmentation includes: (1) Calculate the Laplacian matrix of the similarity matrix W. The calculation formula is L = DW, where D is the degree matrix of the similarity matrix W. That is, add up the elements of each column of the similarity matrix W and put them on the diagonal, and the remaining elements are 0; (2) Calculate the first k eigenvectors u1, ..., u in the generalized eigenvalue problem Lu = λDu k ; with u1,…,u k Construct a matrix U∈R for column vectors n×k ; (3) Let y i is the i-th row vector of matrix U, where i = 1, ..., n; use k-means algorithm to sort the point set (y i ) i＝1，…，n Clustering is performed to obtain clustering results. Different clustering categories represent different rooms in the scene, thus achieving segmentation of the three-dimensional scene.

8. A scene semantic segmentation system, characterized in that: include: The navigation network production module is used to triangulate the input 3D model scene and generate several navigation meshes; The point sampling module is used to form a walkable area by combining the union of several navigation networks, uniformly sampling points in the walkable area to obtain the coordinates of the sample points, and all the sample points constitute the interactive point set of the scene; The shortest path calculation module is used to calculate the simulation path in the navigation grid using the A-Star algorithm and the corner point algorithm, and to calculate the shortest path between each pair of sample points in all interaction point sets; Connectivity value calculation module, used to calculate the connectivity value of each sample point according to the shortest path; The similarity matrix construction module is used to calculate the weight of each edge according to the connectivity value of each sample point, so as to construct the similarity matrix of the undirected graph; Semantic Segmentation Module, which is used to perform scene semantic segmentation on undirected graphs using normalized segmentation.

9. A mobile terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images