A method, device and mobile robot for merging boundaries of segmented regions

Abstract

The present application discloses a method for fusing the boundaries of a segmented region, including matching a first boundary obtained by performing region segmentation based on first map data with a second boundary obtained by performing region segmentation based on second map data, wherein there is a dynamically updated difference between the first map data and the second map data; for the first boundary and the second boundary that are successfully matched, according to the first boundary confidence of the first boundary and the second boundary confidence of the second boundary, fusing the position information of the endpoints of the first boundary with the corresponding endpoints of the second boundary to obtain the position information after the endpoints are fused, and using the boundary determined by the position information after the endpoints are fused as the fused boundary. The present application enables the real-time dynamic region division result to have continuous consistency and stability.

Description

Technical Field

[0001] The present invention relates to the field of machine vision technology, and in particular to a boundary fusion method for segmented areas. Background Art

[0002] Currently, there are two main ways to divide maps in the field of mobile robots. One is to divide based on map graphics, and the other is to divide based on local features. Figure 1 Generally, it is a two-dimensional map. Among them, the graph-based segmentation method is generally used to divide two-dimensional grid maps or two-dimensional topological maps. When constructing maps for three-dimensional point clouds, it is generally necessary to perform a two-dimensional projection and use a plane that is easy to segment as a reference. For the feature-based segmentation method, more often it is through training, combining single-frame data features with representative scenes, or learning features at region switching points, to distinguish different scene areas.

[0003] Traditional image segmentation methods can be used to segment regions in two-dimensional maps, segment objects in two-dimensional images, or segment objects in three-dimensional scenes. These image segmentation methods typically segment regions or objects based on single-shot sampling data. For example, in two-dimensional map segmentation, a global two-dimensional grid map is often used to perform region segmentation.

[0004] Mobile robots often encounter unknown environments, meaning they lack a global map. Collected information, including maps and images, is dynamic and updated in real time. Traditional image segmentation methods, however, produce highly variable results for this dynamic, real-time data, lacking consistency across continuous states. This results in significant variations in real-time segmentation results as the state changes, significantly impacting the mobile robot's strategy selection. Summary of the Invention

[0005] The present invention provides a boundary fusion method for segmented regions to improve the consistency of the segmented regions based on dynamic and real-time updated data.

[0006] The present invention provides a method for merging boundaries of segmented regions, the method comprising:

[0007] matching a first boundary obtained by performing region segmentation based on first map data with a second boundary obtained by performing region segmentation based on second map data, wherein a dynamically updated difference exists between the first map data and the second map data;

[0008] For the first and second boundaries that match successfully,

[0009] According to a first boundary confidence of the first boundary and a second boundary confidence of the second boundary, position information of endpoints of the first boundary and corresponding endpoints of the second boundary are fused to obtain position information after endpoint fusion.

[0010] The boundary determined by the position information after the endpoints are fused is used as the fusion boundary.

[0011] Preferably, the step of fusing the position information of the endpoints of the first boundary and the corresponding endpoints of the second boundary according to the first boundary confidence of the first boundary and the second boundary confidence of the second boundary to obtain the position information after the endpoint fusion includes:

[0012] Using the first boundary confidence and the second boundary confidence, respectively perform weighted fusion on the pixel coordinates of the endpoints of the first boundary and the corresponding endpoints of the second boundary to obtain pixel coordinates after endpoint fusion;

[0013] The boundary confidence is the average of the confidences of the two endpoints on the boundary; wherein the endpoint confidence is: within the endpoint neighborhood, the ratio of the number of consecutive obstacle pixels adjacent to the endpoint to the total number of pixels within the endpoint neighborhood.

[0014] Preferably, the step of using the first confidence level and the second confidence level to weight the pixel coordinates of the endpoints of the first boundary and the corresponding endpoints of the second boundary respectively to obtain pixel coordinates after endpoint fusion includes:

[0015] The ratio of the first boundary confidence to the sum of the first boundary confidence and the second boundary confidence is used as the first fusion weight.

[0016] The ratio of the second boundary confidence to the sum of the first boundary confidence and the second boundary confidence is used as the second fusion weight.

[0017] Multiplying the first fusion weight by the pixel coordinates of the first endpoint on the first boundary to obtain a first result,

[0018] Multiplying the second fusion weight by the pixel coordinate of the first endpoint corresponding to the first endpoint and located on the second boundary to obtain a second result,

[0019] Add the first result and the second result to obtain the pixel coordinates after the first endpoint is fused;

[0020] Multiply the first fusion weight by the pixel coordinates of the second endpoint on the first boundary to obtain a third result,

[0021] Multiplying the second fusion weight by the pixel coordinate of the second endpoint corresponding to the second endpoint and located on the second boundary to obtain a fourth result,

[0022] The third result is added to the fourth result to obtain the pixel coordinates after the second endpoint is fused.

[0023] Preferably, matching the first boundary obtained by performing region segmentation based on the first map data with the second boundary obtained by performing region segmentation based on the second map data comprises:

[0024] For each first boundary,

[0025] If the distance between the centroid of the second boundary and the centroid of the first boundary, and / or the distance between the two endpoints of the second boundary and the corresponding two endpoints of the first boundary are within a set distance threshold, then the second boundary is successfully matched with the first boundary;

[0026] The method further comprises,

[0027] For the first boundary that is not matched successfully, the first boundary confidence of the first boundary is reduced.

[0028] For the second boundary that is not successfully matched, the second boundary is used as the fusion boundary, and the second boundary confidence of the second boundary is kept unchanged;

[0029] For each fused boundary formed by merging the successfully matched first boundary and the second boundary, weighted fusion is performed on the first boundary confidence and the second boundary confidence used for fusion according to the set weights to obtain the boundary confidence of the fused boundary.

[0030] Preferably, reducing the first boundary confidence of the first boundary includes subtracting a set constant from the first boundary confidence of the first boundary,

[0031] The weighted fusion of the first boundary confidence and the second boundary confidence for fusion according to the set weights includes:

[0032] The first boundary confidence is weighted by the set first weight,

[0033] The second boundary confidence is weighted by the set second weight,

[0034] Adding the obtained weighted results;

[0035] The method further comprises:

[0036] Among the fused boundaries formed by merging the successfully matched first boundary and the second boundary, the fused boundaries whose boundary confidence is less than a set confidence threshold are removed.

[0037] Preferably, there is a dynamically updated difference between the first map data and the second map data, including:

[0038] When the difference between the first map data and the second map data is greater than a set difference threshold, determining that there is a dynamically updated difference between the first map data and the second map data;

[0039] The method further comprises,

[0040] When the difference between the first map data and the second map data is greater than a difference threshold, triggering region segmentation based on the first map data;

[0041] The first map data is currently collected map data, and the second map data is last collected map data.

[0042] Preferably, the method further comprises,

[0043] The first map is probabilistically rasterized to obtain a probabilistic raster map, wherein the grid value represents the probability of an obstacle existing in the grid.

[0044] The boundary confidence is the average of the confidences of the two endpoints on the boundary; wherein the endpoint confidence is determined as follows:

[0045] Count the number of continuous obstacle grids adjacent to the endpoint within the endpoint neighborhood, where the obstacle grid is a grid whose grid value is greater than the set probability threshold;

[0046] The ratio of the number of counted grids to the total number of grids in the neighborhood of the endpoint is used as the confidence level of the endpoint.

[0047] Preferably, before matching the first boundary obtained by performing region segmentation based on the first map data with the second boundary obtained by performing region segmentation based on the second map data, the method further includes:

[0048] Load the first partition result after user editing,

[0049] Extract the boundaries edited by the user in the first partition result, mark them, and add them to the user-edited boundary set.

[0050] When the first map data is updated, region segmentation based on the first map data is triggered to obtain a first boundary, add the first boundary to the first boundary set, and calculate a first boundary confidence of each first boundary in the first boundary set;

[0051] According to the mark, the boundary edited by the user in the first boundary set is locked,

[0052] Determine whether there is a second boundary set containing the second boundary,

[0053] If yes, performing the step of matching each first boundary obtained by performing region segmentation based on the first map data with a second boundary obtained by performing region segmentation based on the second map data, and merging the first boundaries in the first boundary set except the boundaries edited by the locked user with the second boundaries in the second boundary set;

[0054] Otherwise, the user-edited boundary set is merged with the first boundary set to form the second boundary set.

[0055] The present application also provides a device for merging boundaries of segmented regions, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the steps of any of the above-mentioned methods for merging boundaries of segmented regions.

[0056] The present application also provides a mobile robot, which includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the steps of any of the above-mentioned boundary fusion methods for segmented areas.

[0057] The present invention provides a boundary fusion method for segmented areas. The method obtains a fused boundary by fusing the position information of the corresponding endpoints of the first and second boundaries obtained by segmenting two map data with dynamically updated differences according to the boundary confidence level. This method solves the problem of inconsistent segmentation results of map data performed by a mobile robot in motion, ensures that the real-time dynamic area division results have continuous consistency and stability, avoids changes in the area segmentation results due to errors and noise in the real-time updated data, and makes the boundary more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of the map boundary divided by traditional graph segmentation methods during the movement of a mobile robot.

[0059] Figure 2 This is a flow chart of boundary fusion based on graphic segmentation results in an embodiment of the present application.

[0060] Figure 3 A schematic diagram of the boundary fusion method for segmented regions.

[0061] Figure 4 A schematic diagram of the partitioning algorithm based on the Voronoi diagram.

[0062] Figure 5a A schematic diagram of a map boundary divided based on the boundary fusion method of this application.

[0063] Figure 5b A schematic diagram of endpoint confidence calculation.

[0064] Figure 6 A schematic diagram of the relationship between the boundaries required in the fusion process and the boundary confidence.

[0065] Figure 7 This is a flow chart of the process of fusing the partition results edited by the user and the partition results divided by the graph segmentation algorithm.

[0066] Figure 8 A schematic diagram of the fusion of the partition results edited by the user and the partition results divided by the graph segmentation algorithm.

[0067] Figure 9 A schematic diagram of a boundary fusion device according to an embodiment of the present application.

[0068] Figure 10 This is another schematic diagram of the boundary fusion device according to an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical means and advantages of this application more clear, this application is further described in detail below with reference to the accompanying drawings.

[0070] The applicant has found that the traditional image segmentation method may cause the boundaries of the segmentation to suddenly disappear during the movement of the mobile robot. As an example, see Figure 1 As shown, Figure 1 This diagram illustrates the map boundaries created using traditional graph segmentation methods during a mobile robot's motion. The circle in the figure represents the mobile robot. In the left figure, when the mobile robot is at the position shown in the left diagram, the segmented area has the boundary defined. However, in the right figure, as the mobile robot moves to the position shown in the right diagram, the boundary suddenly disappears. This is because map data is updated and changed in real time as the mobile robot moves. Especially when the map data is incomplete and contains relatively little information, any local changes to the map significantly affect the overall map distribution, leading to significant discrepancies in the resulting segmentation results.

[0071] This application proposes a confidence-based segmentation and fusion method for real-time dynamic data collection by mobile robots. During the data collection process, the method performs graphic segmentation on each two-dimensional map constructed in real time, calculates its confidence, and fuses the current segmentation result with the previous segmentation result based on the confidence.

[0072] See also Figure 2 As shown, Figure 2 This is a flow chart of a process for performing boundary fusion based on the image segmentation results according to an embodiment of the present application. The method includes:

[0073] Step 201: Match each first boundary in the first boundary set with a second boundary in the second boundary set.

[0074] The first boundary set is a set of first boundaries obtained by performing region segmentation based on the first map data;

[0075] The second boundary set is a set of second boundaries obtained by performing region segmentation based on the second map data;

[0076] There is a dynamically updated difference between the first map data and the second map data;

[0077] Step 202: For the first boundary and the second boundary that are successfully matched, the position information of the endpoints of the first boundary and the corresponding endpoints of the second boundary are fused according to the first boundary confidence of the first boundary and the second boundary confidence of the second boundary to obtain the position information after the endpoint fusion.

[0078] In step 203 , the boundary determined by the position information of the fused endpoints is used as the fused boundary; that is, the line segment determined by the fused positions of the endpoints is the fused boundary.

[0079] This application combines the characteristics of real-time map data updates and achieves continuous segmentation result fusion by fusing the segmentation results of map data with real-time update differences. This results in more robust and accurate boundary information, avoiding the inevitable data fluctuations caused by errors and noise introduced by the real-time data updates of the mobile robot. This effectively solves the corresponding changes in the constructed map and enhances the robustness and consistency of the segmentation results. Fusion based on boundary confidence can produce more robust and temporally consistent segmentation results in real-time scenarios, thereby better meeting the needs of mobile robots in strategy and planning.

[0080] To facilitate understanding of the present application, the following description is made by taking as an example the case where the first map data is the map data currently collected in real time and the second map data is the map data collected last time.

[0081] See also Figure 3 As shown, Figure 3 A schematic diagram of a method for merging the boundaries of segmented regions. For the currently collected and updated two-dimensional map data (first map data), the method for merging the boundaries of segmented regions includes:

[0082] Step 301: determine whether the current map data is updated. If so, trigger the call of the graphic segmentation algorithm to perform graphic segmentation on the current two-dimensional map to obtain segmented areas and boundaries between the segmented areas.

[0083] Preferably, a determination is made as to whether the difference between the currently collected two-dimensional map data (first map) and the previously collected two-dimensional map data (second map) is greater than a set difference threshold. If so, indicating that the currently collected map data has been updated, a graphic segmentation algorithm is triggered to perform graphic segmentation on the current two-dimensional map to obtain segmented regions and first boundaries between the segmented regions. Otherwise, indicating that the currently collected map data has not been updated, the graphic segmentation algorithm is not triggered. The difference may include a difference in the amount of data or a difference in the values ​​of the data itself.

[0084] In this step, the graph segmentation algorithm can adopt a partitioning algorithm based on the Voronoi diagram, which extracts the Voronoi diagram of the binary map, selects key points on the Voronoi diagram, and divides the map into multiple areas through the lines connecting the key points and the adjacent obstacles.

[0085] See also Figure 4 As shown, Figure 4 Figure 1 is a schematic diagram of a partitioning algorithm based on a Voronoi diagram. Figure a represents the input map, Figure b is the extracted Voronoi diagram, and Figure c represents the selection of key points on the Voronoi diagram. These key points are the points in the Voronoi diagram where the local distance to an obstacle is the minimum, where the distance is the Euclidean distance. Figure d represents the area partitioned by the lines connecting the key points and the adjacent obstacles.

[0086] Step 302 : Perform probabilistic rasterization based on the current two-dimensional map data to obtain a probabilistic grid map.

[0087] Probabilistic rasterization divides the map into a grid of a×a cells (the smallest grid). The side length of the grid depends on the map resolution and is usually 1 cm to 5 cm. Each grid cell has a value between 0 and 1.0, which represents the probability that an obstacle exists in that grid cell. A higher value indicates a higher probability that the grid cell is an obstacle.

[0088] Step 303: Determine the boundary confidence based on the probability grid map.

[0089] Taking indoor maps as an example, since areas are usually connected by channels, doors, etc., the two end points of the dividing line should naturally be connected to the edge of the channel. The distribution of obstacles in the neighborhood of the endpoint can be used to determine whether the endpoint is likely to be on the channel or door. The neighborhood range can be set according to the design needs, preferably according to the resolution of the probability grid map. Figure 5a In the left figure, the dotted circle represents the neighborhood range of the endpoint on the boundary. As an example, the radius of the dotted circle is 0.2m, which is converted into the number of pixels according to the map resolution. For example, when the map resolution is 0.05, the neighborhood range of the endpoint is taken as 4 pixels.

[0090] In the probability grid, the probability of an unknown area is 0.5, so grids with values ​​above 0.5 can be regarded as obstacles.

[0091] The first parameter used to measure the confidence level of an endpoint is calculated by counting the number of consecutive obstacle grids adjacent to the endpoint within the endpoint's neighborhood. In other words, the number of grids with values ​​greater than a set probability threshold is counted. Normalizing the first parameter yields the endpoint confidence level. Alternatively, dividing the first parameter by the total number of grids within the endpoint's neighborhood yields the endpoint confidence level. This can be expressed mathematically as:

[0092] e=y1 / y2

[0093] Among them, e is the endpoint confidence; y1 is the number of continuous obstacle grids adjacent to the endpoint in the neighborhood range of the endpoint, that is, the first parameter; y2 is the number of grids in the neighborhood range.

[0094] For example, Figure 5b Grid 1 represents the endpoint of the boundary, the black grid represents the obstacle, and the dashed box represents the obstacle in the neighborhood included in the statistics. The black grid 2 in the lower left corner is not adjacent to the endpoint and is not a continuous obstacle point adjacent to the endpoint in the neighborhood, so it is not included in the statistics. Therefore, the first parameter of the endpoint in the figure is 6. If the neighborhood range is 6×6, the confidence level of this endpoint is 6 / (6×6).

[0095] Given that a probabilistic grid map can be generated based on the desired map resolution, the number of pixels processed can be reduced. Therefore, calculating endpoint confidence based on the probabilistic grid map can improve processing speed. It should be understood that endpoint confidence can also be directly processed based on pixel information from the image map. Thus, the endpoint confidence can be calculated as the ratio of the number of consecutive obstacle pixels adjacent to the endpoint within the endpoint's neighborhood to the total number of pixels within the endpoint's neighborhood.

[0096] A boundary has two endpoints, and the confidence of the two endpoints is averaged and recorded as the confidence of the boundary.

[0097] Step 304: Determine whether there is a second boundary set obtained from the previous segmentation and the boundary confidence of each second boundary in the set. If so, execute step 305. Otherwise, use the current first boundary set as the second boundary set and the current first boundary confidence set as the second boundary confidence set, and then execute step 307.

[0098] Step 305: Fusing the boundaries according to the boundary confidences, and updating the boundary confidences of the fused boundaries.

[0099] Let the first boundary set obtained by segmentation based on the current map data be the first boundary set, denoted as B, bi is any first boundary in the first boundary set, and the boundary confidence set of the first boundary is C, b i The corresponding boundary confidence is c i , and the second boundary set is B L , the boundary confidence set of the second boundary is C L The final fusion boundary set is B new , this set is the set of fused boundaries, and the boundary confidence set of the fused boundaries is C new .

[0100] For each first boundary b in the first boundary set B i , in the second boundary set B L Find the corresponding second boundary in The search condition is: the distance between the centroids of the two boundaries and / or the corresponding two end points is within the set distance threshold. If the corresponding boundary is found, the corresponding relationship between the two boundaries is recorded and recorded as the first boundary b. n and the second boundary Corresponding. Thus, the first boundary set B and the second boundary set B L The boundaries in can include the boundaries that are matched successfully and the boundaries that are not matched successfully.

[0101] For successfully matched boundaries, that is, for the first and second boundaries that have been successfully matched, boundary fusion is performed. Specifically, the fusion method may be to determine a fusion weight based on the boundary confidence, use the fusion weight to calculate the endpoint position information of the fused boundary after fusion, and use the boundary determined by the calculated endpoint position information as the fused boundary.

[0102] Let the two endpoints of the first boundary bn be and Corresponding to the second boundary The two endpoints in Know The fused boundary The two endpoints and They are:

[0103]

[0104]

[0105]

[0106]

[0107] Among them, λ n is the first fusion weight, is the second fusion weight, cn The first boundary b n The first bound confidence level of The second boundary In this way, it can be understood that the first boundary confidence and the second boundary confidence are used to obtain a fusion weight, and the fusion weight is used to perform weighted fusion on the pixel coordinates of the endpoints of the first boundary and the corresponding endpoints of the second boundary.

[0108] Expand to calculate the pixel coordinates of the endpoints:

[0109] Multiplying the first fusion weight by the pixel coordinates of the first endpoint on the first boundary to obtain a first result,

[0110] Multiplying the second fusion weight by the pixel coordinate of the first endpoint corresponding to the first endpoint and located on the second boundary to obtain a second result,

[0111] Add the first result and the second result to obtain the pixel coordinates after the first endpoint is fused;

[0112] Multiply the first fusion weight by the pixel coordinates of the second endpoint on the first boundary to obtain a third result,

[0113] Multiplying the second fusion weight by the pixel coordinate of the second endpoint corresponding to the second endpoint and located on the second boundary to obtain a fourth result,

[0114] The third result is added to the fourth result to obtain the pixel coordinates after the second endpoint is fused.

[0115] It can be expressed mathematically as:

[0116]

[0117]

[0118]

[0119]

[0120] Among them, the endpoint The pixel coordinates are endpoint The pixel coordinates are endpoint The pixel coordinates are endpoint The pixel coordinates are endpoint The pixel coordinates are endpoint The pixel coordinates are

[0121] Blending Boundaries Boundary confidence Determine as follows:

[0122] The first boundary confidence is weighted by the set first weight,

[0123] The second boundary confidence is weighted by the set second weight,

[0124] Adding the obtained weighted results;

[0125] It can be expressed mathematically as:

[0126]

[0127] Here, λ1 and λ2 are the first and second weights, respectively. They can be set based on experience. The sum of the first and second weights is 1. The size of the two weights indicates whether the confidence level of the boundary of the current segmentation or the boundary of the previous segmentation is higher. When the confidence levels are the same, the first and second weights are both 0.5.

[0128] For the first boundary in the first boundary set B that is not matched successfully, since no boundary matching the first boundary b is found in the second boundary set, k The corresponding second boundary is lowered by the first boundary b k First bound confidence Specifically, they can be:

[0129]

[0130] in, is the first boundary confidence after reduction, c k The first boundary b k The original first boundary confidence, c const is a set constant.

[0131] For the second boundary set B L The second boundary in the unsuccessful match Then directly add the second boundary to the fusion boundary set B new In order to use the second boundary as the fusion boundary and keep the second boundary confidence of the second boundary unchanged, it can be expressed mathematically as follows:

[0132]

[0133]

[0134] Step 306 , for the fused boundaries formed by the successfully matched first and second boundaries in the fused boundary set, delete the fused boundaries whose boundary confidence is less than the set confidence threshold. For example, delete the boundaries whose boundary confidence is 0 so that the fused boundaries are no longer used as boundaries.

[0135] Step 307 , determining whether the current map data collection is completed, if so, then ending the process, otherwise, returning to step 301 .

[0136] See also Figure 6 As shown, Figure 6 It is a schematic diagram of the relationship between the boundaries required in the fusion process and the boundary confidence. L , fusion boundary set B new , first boundary confidence set C, second boundary confidence set C L , the boundary confidence set C of the fusion boundary new , the first boundary b that is successfully matched n , the second boundary of successful matching The first boundary bk that was not matched successfully, the second boundary that was not matched successfully A relationship between.

[0137] From the perspective of the boundary fusion process, fusion is related to the successfully matched boundaries and their boundary confidence, and has nothing to do with the way the map data is collected. In this way, the boundary fusion method of this application is not limited to the fusion of graphic segmentation results from map data collected by the same sensor, but can also be used to fuse graphic segmentation results from map data collected by multiple sensors. For example, the regional division boundaries of the raster map constructed by radar and the boundaries divided by door frame recognition in vision can also be boundary fused. When used for boundary fusion under multiple sensors, it is only necessary to normalize the boundary confidence.

[0138] The boundary fusion method of the embodiment of the present application fuses the first boundary obtained based on the first map segmentation and the second boundary obtained based on the second map segmentation according to the boundary confidence, so that the real-time dynamic area division result has continuous consistency and stability, and there will be no sudden changes or instability in the strategy of the mobile robot in judging its own area and planning the area path due to the intermittent division boundary caused by the movement of the mobile robot. Figure 5a In the example, after adopting the fusion method of the present application, even if the position of the mobile robot changes, the boundary will not change. Figure 1 The situation where the middle boundary suddenly disappears, Figure 5aThe boundary information in the image is more stable. Furthermore, during the continuous movement of the mobile robot, whenever the difference between the first and second map data exceeds a set threshold, the current segmentation boundary is fused. This achieves the fusion of the segmentation results in the mobile robot's continuous state, making the area segmentation results closer to the actual results and more reliable.

[0139] Furthermore, by applying the boundary fusion method of the present application, it is also possible to achieve the fusion of the partition results edited by the user and the partition results divided by the graphic segmentation algorithm. As an example, for traditional general algorithms, the partition results will remain unchanged after user intervention. However, this approach cannot produce corresponding operations when new changes occur in the scene, and the user needs to intervene in the division again. By using the boundary fusion method of the present application, the scene can be further divided based on the user's editing, which can better cope with changes in the scene and is more intelligent.

[0140] See also Figure 7 As shown, Figure 7 This is a flow chart of the fusion of the partition results edited by the user and the partition results divided by the graph segmentation algorithm. The fusion method includes:

[0141] Step 701: Load the first partition result edited by the user.

[0142] Step 702: extract the boundaries edited by the user in the first partition result, mark them, and add the boundaries edited by the user to the user edited boundary set, which is recorded as B. u , to distinguish the boundary edited by the user from the boundary area obtained by the graphic segmentation algorithm,

[0143] As an example, a boundary marker edited by the user can be set to locked and saved as a property.

[0144] Step 703, determining whether the current map data (first map data) is updated, if so, executing step 704, otherwise, returning to step 703;

[0145] Step 704: trigger the graph segmentation algorithm to perform region segmentation based on the current map data to obtain the current boundary set B. a (first boundary set), and calculate the boundary confidence of the current boundary (first boundary), and obtain the boundary confidence set C of the current boundary a ,

[0146] Step 705: Based on the set attribute mark, a Lock the boundary edited by the user to prevent fusion of the boundary edited by the user;

[0147] Step 706: Determine whether there is a second boundary set B L And its boundary confidence set C L ,

[0148] If yes, then according to step 305 or step 201, the current boundary set B a The boundaries except the boundaries edited by the locked user and the second boundary set B L The boundaries in the , are merged, so the boundaries edited by the user are not merged.

[0149] Otherwise, the user edits the boundary set B u With the current boundary set B a Merge to add the user-edited boundaries to the current boundary set B a As the second boundary set B L .

[0150] Step 707 , determining whether map data collection is completed, if so, then terminate this process, otherwise, return to step 703 .

[0151] See also Figure 8 As shown, Figure 8 A schematic diagram of the fusion of the partitioning results edited by the user and the partitioning results divided by the graphic segmentation algorithm. In the figure, a is the original map used by the user for editing, b is the user editing result, c is the map after the scene changes, and d is the result of real-time dynamic segmentation based on the user editing result. It can be seen from c and d that after the map is updated, the part of the user editing result has not changed, while the updated part of the map is divided. Since the updated map can be segmented on the basis of retaining the user editing result, the user does not need to re-divide the updated map. In addition, the user's markings for the edited areas can be retained, such as the kitchen, living room, bedroom, etc., which will continue to be used in the new segmentation results, and the user does not need to re-mark them again. It can be seen that through the fusion method of the present application, combined with the user's partition editing results, the scene segmentation can be updated based on the user's editing, which has better applicability to scene changes and reduces the invalid operations of the user's repeated editing of the updated scene.

[0152] See also Figure 9 As shown, Figure 9 This is a schematic diagram of a boundary fusion device according to an embodiment of the present application. The device includes:

[0153] a matching module, configured to match each first boundary in the first boundary set with a second boundary in the second boundary set,

[0154] a boundary fusion module configured to fuse, for a successfully matched first boundary and second boundary, position information of endpoints of the first boundary with corresponding endpoints of the second boundary based on a first boundary confidence of the first boundary and a second boundary confidence of the second boundary, to obtain position information of the fused endpoints, and use the boundary determined by the fused endpoint position information as the fused boundary;

[0155] Preferably, the device further comprises:

[0156] A region segmentation module, configured to trigger graphic segmentation to obtain a first boundary when the first map data is updated;

[0157] Confidence calculation module, used to calculate boundary confidence,

[0158] The confidence calculation module includes:

[0159] The endpoint confidence calculation submodule is used to calculate the endpoint confidence of the endpoint on the boundary.

[0160] The boundary confidence calculation submodule is used to calculate the boundary confidence based on the endpoint confidence.

[0161] See also Figure 10 As shown, Figure 10 This is another schematic diagram of a boundary fusion device according to an embodiment of the present application, comprising a memory and a processor, wherein the memory stores a computer program and the processor is configured to execute the steps of the boundary fusion method for segmented regions.

[0162] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0163] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0164] An embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the boundary fusion method of the segmented regions are implemented.

[0165] As for the apparatus / network-side device / storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0166] In this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A boundary fusion method for segmented regions, characterized in that: The method includes, matching a first boundary obtained by performing region segmentation based on first map data with a second boundary obtained by performing region segmentation based on second map data, wherein a dynamically updated difference exists between the first map data and the second map data; For the first and second boundaries that match successfully, According to a first boundary confidence of the first boundary and a second boundary confidence of the second boundary, position information of endpoints of the first boundary and corresponding endpoints of the second boundary are merged to obtain position information after endpoint fusion, The boundary determined by the position information after the endpoints are fused is used as the fusion boundary; in, The boundary confidence is the average of the confidences of the two endpoints on the boundary. The endpoint confidence is the ratio of the number of continuous obstacle pixels adjacent to the endpoint within the neighborhood of the endpoint to the total number of pixels within the neighborhood of the endpoint.

2. The method according to claim 1, characterized in that The step of fusing the position information of the endpoints of the first boundary and the corresponding endpoints of the second boundary according to the first boundary confidence of the first boundary and the second boundary confidence of the second boundary to obtain the position information after the endpoints are fused includes: The first boundary confidence and the second boundary confidence are used to perform weighted fusion on the pixel coordinates of the endpoints of the first boundary and the corresponding endpoints of the second boundary, respectively, to obtain pixel coordinates after endpoint fusion.

3. The method according to claim 2, characterized in that The step of using the first confidence level and the second confidence level to respectively weightedly fuse the pixel coordinates of the endpoints of the first boundary and the endpoints corresponding to the second boundary to obtain pixel coordinates after endpoint fusion includes: The ratio of the first boundary confidence to the sum of the first boundary confidence and the second boundary confidence is used as the first fusion weight. The ratio of the second boundary confidence to the sum of the first boundary confidence and the second boundary confidence is used as the second fusion weight. The first fusion weight is multiplied by the pixel coordinates of the first endpoint on the first boundary to obtain a first result, The second fusion weight is multiplied by the pixel coordinate of the first endpoint corresponding to the first endpoint and located on the second boundary to obtain a second result, Add the first result and the second result to obtain the pixel coordinates after the first endpoint is fused; The first fusion weight is multiplied by the pixel coordinate of the second endpoint on the first boundary to obtain a third result, The second fusion weight is multiplied by the pixel coordinate of the second endpoint corresponding to the second endpoint and located on the second boundary to obtain a fourth result, The third result is added to the fourth result to obtain the pixel coordinates after the second endpoint is merged.

4. The method according to claim 1, characterized in that The matching of a first boundary obtained by performing region segmentation based on the first map data with a second boundary obtained by performing region segmentation based on the second map data comprises: For each first boundary, If the distance between the centroid of the second boundary and the centroid of the first boundary, and / or the distance between the two end points of the second boundary and the corresponding two end points of the first boundary are within a set distance threshold, then the second boundary is successfully matched with the first boundary; The method further comprises, For the first boundary that is not matched successfully, the first boundary confidence of the first boundary is reduced. For the second boundary that is not matched successfully, the second boundary is used as the fusion boundary, and the second boundary confidence of the second boundary is kept unchanged; For each fused boundary formed by merging the successfully matched first boundary and the second boundary, weighted fusion is performed on the first boundary confidence and the second boundary confidence used for fusion according to the set weights to obtain the boundary confidence of the fused boundary.

5. The method according to claim 4, characterized in that The step of reducing the first boundary confidence of the first boundary comprises: subtracting a set constant from the first boundary confidence of the first boundary, The weighted fusion of the first boundary confidence and the second boundary confidence used for fusion according to the set weights includes: The first boundary confidence is weighted by the set first weight, The second boundary confidence is weighted by the set second weight, Add the obtained weighted results; The method further comprises: Among the fused boundaries formed by merging the successfully matched first boundary and the second boundary, the fused boundaries whose boundary confidence is less than a set confidence threshold are removed.

6. The method according to claim 1, characterized in that There is a dynamically updated difference between the first map data and the second map data, include, When the difference between the first map data and the second map data is greater than a set difference threshold, determining that there is a dynamically updated difference between the first map data and the second map data; The method further comprises, When the difference between the first map data and the second map data is greater than a difference threshold, triggering region segmentation based on the first map data; The first map data is currently collected map data, and the second map data is last collected map data.

7. The method according to claim 6, characterized in that The method further comprises, The first map is probability gridded to obtain a probability grid map, wherein the grid value represents the probability of the existence of an obstacle in the grid, The boundary confidence is the average of the confidences of the two end points on the boundary; wherein the endpoint confidence is determined as follows: Count the number of continuous obstacle grids adjacent to the endpoint within the neighborhood of the endpoint, where the obstacle grid is a grid whose grid value is greater than a set probability threshold; The ratio of the number of counted grids to the total number of grids in the neighborhood of the endpoint is taken as the confidence of the endpoint.

8. The method according to claim 1, characterized in that Before matching the first boundary obtained by performing region segmentation based on the first map data with the second boundary obtained by performing region segmentation based on the second map data, the method further includes: Load the first partition result after user editing, Extract the boundary edited by the user in the first partition result, mark it, and add the boundary edited by the user to the user edit boundary set. When the first map data is updated, region segmentation based on the first map data is triggered to obtain a first boundary, add the first boundary to the first boundary set, and calculate a first boundary confidence of each first boundary in the first boundary set; According to the mark, locking the boundary edited by the user in the first boundary set, Determine whether there is a second boundary set that includes the second boundary, If yes, the step of matching each first boundary obtained by performing region segmentation based on the first map data with a second boundary obtained by performing region segmentation based on the second map data is performed, and the first boundaries in the first boundary set except the boundaries edited by the locked user are merged with the second boundaries in the second boundary set; Otherwise, the user-edited boundary set is merged with the first boundary set to form the second boundary set.

9. A device for merging boundaries of segmented regions, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the steps of the boundary fusion method for segmented regions according to any one of claims 1 to 8.

10. A mobile robot, characterized in that: The mobile robot comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the steps of the boundary fusion method for segmented areas according to any one of claims 1 to 8.

Images