Region segmentation and autonomous target search method based on environmental semantic information

Abstract

The invention discloses a region segmentation and autonomous target searching method based on environmental semantic information, and belongs to the technical field of robot autonomous navigation and environmental perception. Comprising the steps of performing region division on a to-be-searched region based on environment semantic information, and constructing a map; for each region, detecting an object in the region in real time by using a deep learning model, and obtaining position coordinate information of the object, an object name and a category of the region to which the object belongs; counting the frequencies of different types of objects in each region, and naming the current region according to the type of the object with the highest occurrence frequency; and according to the name of a to-be-searched object in the target search task, corresponding to the region category to which the to-be-searched object belongs, performing target search based on the map. According to the method, automatic region naming and accurate segmentation based on object categories are realized by simulating a human cognitive mode, dynamic task adjustment and efficient object searching are supported at the same time, and the navigation efficiency and the reliability of target positioning are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of robot autonomous navigation and environment perception, and in particular to a method for region segmentation and autonomous target search based on environmental semantic information. Background Art

[0002] With the rapid development of robotics, there is a growing demand for autonomous navigation methods such as home service robots. Traditional navigation methods usually rely on fixed area classification strategies and predefined maps, which show obvious limitations when facing dynamic or unknown environments. Especially in home environments, where area layouts are diverse and object distribution is complex, robots need to have more flexible environmental perception and area recognition capabilities to complete tasks efficiently.

[0003] Most existing region segmentation technologies rely on geometric features for division, such as obtaining spatial information through lidar, depth sensors or image processing methods. These methods can divide physical space, but it is difficult to associate the functional attributes of the region, resulting in low navigation efficiency in complex scenes. For example, traditional methods have difficulty distinguishing between areas with similar functions but different spatial layouts (such as living rooms and study rooms), or accurately locating the area to which specific items belong. This takes into account that humans will quickly judge the function of an area by observing the category of objects in an unfamiliar environment. For example, seeing a sofa and a TV will naturally classify the area as a living room. This intuitive behavior pattern enables humans to quickly adapt and locate target objects. The region division and navigation strategy based on this behavior pattern can make up for the shortcomings of existing technologies and enable robots to achieve more intelligent navigation and task execution in dynamic environments. Summary of the invention

[0004] In response to the shortcomings of the existing technology, the present invention introduces a region recognition and segmentation strategy based on environmental semantic information, combined with deep learning and depth perception technology, to achieve human-like region judgment, automatic naming and dynamic task switching, greatly improving the robot's navigation efficiency and object-finding accuracy in scenarios such as home services and warehouse management.

[0005] In order to achieve the above object, the present invention provides a method for region segmentation and autonomous target search based on environmental semantic information, comprising the following steps:

[0006] (1) Divide the search area into regions based on environmental semantic information and construct a map;

[0007] (2) for each of the regions, using a deep learning model to detect objects in the region in real time, and obtaining the location coordinate information of the object, the name of the object, and the category of the region to which the object belongs;

[0008] (3) counting the frequency of objects of different categories in each of the regions, and naming the current region according to the object category with the highest frequency;

[0009] (4) According to the name of the object to be found in the target search task, corresponding to the area category to which it belongs, the target search is performed based on the map.

[0010] Furthermore, it is characterized in that the step (1) specifically comprises:

[0011] (1.1) Start the robot and move forward in a straight line until the road ahead is impassable. The robot takes the current position as the initial point and turns left to move forward.

[0012] (1.2) The robot detects in real time whether the left and right sides of the robot are passable;

[0013] When it is detected that the right side is passable, turn right and move forward;

[0014] When it is detected that both the left and right sides are impassable, this is a door. The current posture is compared with the saved posture at the same position. If the posture is opposite, the area is established and recorded as the area entrance. If there is no saved posture, the current posture is saved and the vehicle continues to move forward.

[0015] When it is detected that the left side is passable but the right side is not, check whether the front is passable. If the front is passable, continue to move forward, otherwise turn left and move forward;

[0016] (1.3) Repeat step (1.2) until you return to the initial point and complete the establishment of the area where the initial point is located;

[0017] (1.4) Save the coordinates of each area and complete the map construction.

[0018] Furthermore, the robot detects whether the front, left, or right sides are passable based on the laser radar data on the robot, specifically:

[0019] Clean the lidar data and remove invalid data;

[0020]

[0021] Among them, D min and D max are the minimum and maximum detection ranges of the LiDAR, respectively;

[0022] In the sector range in front of the robot, calculate the average value d of the front depth data avg1 ;

[0023]

[0024] Where: N1 is the number of valid depth data points in the front sector; D valid (θ) is the effective depth data after cleaning; θ1 is the first preset angle;

[0025] If the average distance d avg1 Exceeds the preset front area detection threshold d threshold1 , then it is determined to be passable, otherwise it is impassable;

[0026] Within the left sector of the robot, calculating the average value of the left depth data;

[0027]

[0028] Where: N2 is the number of valid depth data points in the left sector; θ2 is the second preset angle;

[0029] Average distance d avg2 Exceeds the preset detection threshold d of the measurement area threshold2 , then it is determined to be passable, otherwise it is impassable;

[0030] Within the right sector of the robot, calculating the average value of the left depth data;

[0031]

[0032] Where: N3 is the number of valid depth data points in the right sector;

[0033] Average distance d avg3 Exceeds the preset detection threshold d of the measurement area threshold2 , it is determined to be passable, otherwise it is impassable.

[0034] Furthermore, the step (2) is specifically as follows:

[0035] (2.1) Use the depth camera on the robot to obtain RGB images and depth images of different objects in each area;

[0036] (2.2) Using a deep learning model to obtain the center point of the object in the RGB image, and extracting the object name based on the COCO dataset;

[0037] (2.3) obtaining the category of the region to which the object belongs according to the name of the object;

[0038] (2.4) Acquire a depth value of the object by combining the center point information of the object with the depth image;

[0039] (2.5) Obtain the robot coordinates based on the radar information and combine it with the object depth value to obtain the object coordinates;

[0040] (2.6) Save the location coordinates, object names and categories of the areas to which all objects in each area belong.

[0041] Furthermore, the step (2.5) is specifically as follows:

[0042] (2.5.1) calculating the pixel position of the robot in the map;

[0043]

[0044] Where: (x map ,y map ) is the current position of the robot, (x origin ,y origin ) is the position corresponding to the origin of the map, obtained according to the radar data; (u, v) is the pixel position of the robot in the map; Resolution is the map resolution;

[0045] (2.5.2) obtaining the coordinates of the object in the map according to the pixel position of the robot in the map, the center point information of the object and the depth value of the object;

[0046]

[0047] Z c =d

[0048] Where: (c x 、c y ) is the center point information of the object; d is the depth value of the object; (X c , Y c , Z c ) is the coordinate of the object in the map; f x 、f y is the focal length of the camera;

[0049] (2.5.2) Convert to the world coordinate system to obtain the position coordinates of the object;

[0050]

[0051] Where: P w is the coordinate of the object; R is the rotation matrix, and t is the translation vector, which are obtained by jointly calibrating the camera and radar.

[0052] Furthermore, the target search task includes a main task and a secondary task, and the specific search process is as follows:

[0053] (4.1) According to the name of the object to be found in the main task, determine the area category to which the object to be found in the main task belongs; if the sub-task is not empty, determine the area category to which the object to be found in the sub-task belongs according to the name of the object to be found in the sub-task;

[0054] (4.2) The robot enters the area of ​​the object to be found in the primary task according to the map to search for the object. If the object to be found in the secondary task is detected during the process, the robot dynamically switches to the secondary task and automatically resumes the primary task after completing the secondary task.

[0055] (4.3) When the main task is completed, the target search task is completed.

[0056] The present invention also provides a region segmentation and autonomous target search system based on environmental semantic information, comprising:

[0057] A map construction module, which is used to divide the search area into regions based on environmental semantic information and construct a map;

[0058] A detection module is used to detect objects in each area in real time using a deep learning model to obtain location coordinate information and physical categories of the objects;

[0059] The region naming module is used to count the frequency of objects of different categories in each region and name the current region according to the object category with the highest frequency;

[0060] The target search module is used to perform target search based on the map according to the name of the object to be found in the target search task and the corresponding area category to which it belongs.

[0061] Beneficial effects of the present invention:

[0062] The present invention improves the intelligence of robots in area division, object recognition and task management, especially by simulating human cognitive patterns to achieve automatic naming and precise segmentation of areas based on object categories, while supporting dynamic task adjustment and efficient object search. Compared with traditional methods, the present invention abandons the limitations of fixed area settings and static path planning, and through deep learning models and depth perception technology, enables robots to flexibly adapt in unknown environments, and significantly improves navigation efficiency and target positioning reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a flowchart of a method for region segmentation and autonomous target search based on environmental semantic information according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The present invention is described in detail below in conjunction with the accompanying drawings and embodiments. This embodiment is implemented based on the technical solution of the present invention and provides a detailed operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0065] like Figure 1As shown, the present invention provides a method for region segmentation and autonomous target search based on environmental semantic information, comprising the following steps:

[0066] S101. Divide the search area into regions based on environmental semantic information and construct a map.

[0067] The embodiment of the present invention adopts the strategy of "right priority" based on human behavior to divide the area and complete the map construction, specifically:

[0068] (1) Start the robot and move forward in a straight line until the road ahead becomes impassable. The robot then turns left and moves forward, using the current position as the initial point.

[0069] (2) The robot detects in real time whether the left and right sides of the robot are passable.

[0070] When it is detected that the right side is passable, turn right and move forward;

[0071] When it is detected that both the left and right sides are impassable, this is a door. The current posture is compared with the saved posture at the same position. If the posture is opposite, the area is established and recorded as the area entrance. If there is no saved posture, the current posture is saved and the vehicle continues to move forward.

[0072] When it is detected that the left side is passable but the right side is not, check whether the front is passable. If the front is passable, continue to move forward, otherwise turn left and move forward;

[0073] (3) Repeat step (2) until returning to the initial point, completing the establishment of the area where the initial point is located.

[0074] At this point, all contours of the area have been walked by the robot.

[0075] (4) Save the coordinates of each area and complete the map construction.

[0076] The robot detects whether the front, left, or right sides are passable based on the laser radar data on the robot. Specifically:

[0077] Clean the lidar data and remove invalid data.

[0078]

[0079] Among them, D min and D max are the minimum and maximum detection ranges of the LiDAR, respectively.

[0080] In the sector in front of the robot, calculate the average value d of the front depth data avg1 .

[0081]

[0082] Where: N1 is the number of valid depth data points in the front sector; D valid (θ) is the effective depth data after cleaning; θ1 is the first preset angle.

[0083] If the average distance d avg1 Exceeds the preset front area detection threshold d threshold1 , it is determined to be passable, otherwise it is impassable.

[0084] Within the left sector of the robot, the average value of the left depth data is calculated.

[0085]

[0086] Wherein: N2 is the number of valid depth data points in the left sector; θ2 is the second preset angle.

[0087] Average distance d avg2 Exceeds the preset detection threshold d of the measurement area threshold2 , it is determined to be passable, otherwise it is impassable.

[0088] Within the right sector of the robot, the average value of the left depth data is calculated.

[0089]

[0090] Where: N3 is the number of valid depth data points in the right sector.

[0091] Average distance d avg3 Exceeds the preset detection threshold d of the measurement area threshold2 , it is determined to be passable, otherwise it is impassable.

[0092] The coordinates of each area are determined based on the AMCL (Adaptive Monte Carlo Localization) of the robot ROS and saved in the map file.

[0093] The embodiment of the present invention determines whether the left and right sides are passable and saves the coordinates of the entrances of each area based on radar information.

[0094] S102: For each of the regions, use a deep learning model to detect objects in the region in real time, and obtain the location coordinate information of the object, the name of the object, and the category of the region to which the object belongs.

[0095] (1) Use the depth camera on the robot to obtain RGB images and depth images of different objects in each area.

[0096] Through the depth camera installed on the robot, RGB images and depth images of different objects in each area are obtained in real time.

[0097] (2) Use a deep learning model to obtain the center point of the object in the RGB image and extract the object name based on the home environment dataset.

[0098] The home environment dataset is pre-constructed based on the names of all possible objects in the home environment. In the embodiment of the present invention, the COCO (Common Objects in Context) dataset is used as the home environment dataset.

[0099] The COCO dataset is a large-scale dataset for object detection, segmentation, and captioning tasks. It contains more than 330,000 images, each annotated with 80 object categories and 5 captions describing scenes. The COCO dataset is widely used in computer vision research and has been used to train and evaluate many state-of-the-art object detection and segmentation models.

[0100] The embodiment of the present invention is based on the YOLOv8 deep learning model and obtains the center point and object name of the object in the RGB image based on the COCO dataset.

[0101] (3) Obtaining the category of the area to which the object belongs according to the name of the object.

[0102] According to the object name obtained in step (2), determine the category of the area to which it belongs. The categories of the area to which the object belongs include living room, bedroom, kitchen, bathroom, study, etc. This can be pre-matched for the 80 objects in the COCO dataset.

[0103] (4) Combine the object center point information with the depth image to obtain the depth value of the object.

[0104] (5) Obtain the robot coordinates based on the radar information and combine them with the object depth value to obtain the object coordinates.

[0105] (5.1) Calculating the pixel position of the robot in the map;

[0106]

[0107]

[0108] Where: (x map ,y map ) is the current position of the robot, (x origin ,y origin) is the position corresponding to the origin of the map, which is determined by the robot's radar, odometer, and inertial measurement unit based on ROS's AMCL (Adaptive Monte Carlo Localization); (u, v) are the pixel positions of the robot in the map; Resolution is the map resolution.

[0109] (5.2) obtaining the coordinates of the object in the map according to the pixel position of the robot in the map, the center point information of the object and the depth value of the object;

[0110]

[0111] Z c =d

[0112] Where: (c x 、c y ) is the center point information of the object, d is the depth value of the object, (X c , Y c , Z c ) is the coordinate of the object in the map; f x 、f y is the focal length of the camera;

[0113] (5.3) Convert to the world coordinate system to obtain the position coordinates of the object;

[0114]

[0115] Where: P w is the coordinate of the object; R is the rotation matrix, and t is the translation vector, which are obtained by jointly calibrating the camera and radar.

[0116] (6) Save the location coordinates, object names and categories of the areas to which all objects in each area belong.

[0117] For each area, the location coordinates, object names and categories of the areas to which the objects belong of all objects in the area are saved in the map file.

[0118] S103: Count the frequencies of objects of different categories in each of the regions, and name the current region according to the category of objects with the highest frequency of occurrence.

[0119] Save the names of each area to the map file.

[0120] S104, performing target search based on the map according to the name of the object to be found in the target search task and the corresponding area category to which it belongs.

[0121] In the embodiment of the present invention, the target search task may include multiple tasks at the same time, which are divided into a main task and a secondary task. The specific process is as follows:

[0122] (1) According to the name of the object to be found in the main task, determine the area category to which the object to be found in the main task belongs; if the sub-task is not empty, then according to the name of the object to be found in the sub-task, determine the area category to which the object to be found in the sub-task belongs.

[0123] (2) The robot navigates into the area where the object to be found of the primary task is located to search. If the object to be found of the secondary task is detected during this process, the robot dynamically switches to the secondary task and automatically resumes the primary task after completing the secondary task.

[0124] If the object to be found is included in the objects saved in the map file, you can directly locate the object and search for it. If it is not saved in the objects saved in the map, navigate to the area where it belongs to search for it.

[0125] (3) When the main task is completed, the target search task ends.

[0126] Once the main task is completed, the secondary task will become invalid immediately.

[0127] The embodiment of the present invention further provides a region segmentation and autonomous target search system based on environmental semantic information, comprising:

[0128] The map construction module is used to divide the search area into regions based on environmental semantic information and construct a map.

[0129] The detection module is used to detect objects in each area in real time using a deep learning model to obtain the location coordinate information and physical category of the objects.

[0130] The region naming module is used to count the frequency of objects of different categories in each region and name the current region according to the object category with the highest frequency.

[0131] The target search module is used to search for targets based on the map according to the name of the object to be found in the target search task and the corresponding area category to which it belongs.

[0132] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for region segmentation and autonomous target search based on environmental semantic information, characterized in that: The steps include: (1) Divide the search area into regions based on environmental semantic information and construct a map; (2) for each of the regions, using a deep learning model to detect objects in the region in real time, and obtaining the location coordinate information of the object, the name of the object, and the category of the region to which the object belongs; (3) counting the frequency of objects of different categories in each of the regions, and naming the current region according to the object category with the highest frequency; (4) According to the name of the object to be found in the target search task, corresponding to the area category to which it belongs, the target search is performed based on the map.

2. The method for region segmentation and autonomous target search based on environmental semantic information according to claim 1, characterized in that: The step (1) is specifically: (1.1) Start the robot and move forward in a straight line until the road ahead is impassable. The robot takes the current position as the initial point and turns left to move forward. (1.2) The robot detects in real time whether the left and right sides of the robot are passable; When it is detected that the right side is passable, turn right and move forward; When it is detected that both the left and right sides are impassable, this is a door. The current posture is compared with the saved posture at the same position. If the posture is opposite, the area is established and recorded as the area entrance. If there is no saved posture, the current posture is saved and the vehicle continues to move forward. When it is detected that the left side is passable but the right side is not, check whether the front is passable. If the front is passable, continue to move forward, otherwise turn left and move forward; (1.3) Repeat step (1.2) until you return to the initial point and complete the establishment of the area where the initial point is located; (1.4) Save the coordinates of each area and complete the map construction.

3. The method for region segmentation and autonomous target search based on environmental semantic information according to claim 2, characterized in that: The robot detects whether the front, left, or right sides are passable based on the laser radar data on the robot, specifically: Clean the lidar data and remove invalid data; Among them, D min and D max are the minimum and maximum detection ranges of the LiDAR, respectively; In the sector range in front of the robot, calculate the average value d of the front depth data avg1 ; Where: N1 is the number of valid depth data points in the front sector; D valid (θ) is the effective depth data after cleaning; θ1 is the first preset angle; If the average distance d avg1 Exceeds the preset front area detection threshold d threshold1 , then it is determined to be passable, otherwise it is impassable; Within the left sector of the robot, calculating the average value of the left depth data; Where: N2 is the number of valid depth data points in the left sector; θ2 is the second preset angle; the average distance d avg2 Exceeds the preset detection threshold d of the measurement area threshold2 , then it is determined to be passable, otherwise it is impassable; Within the right sector of the robot, calculating the average value of the left depth data; Where: N3 is the number of valid depth data points in the right sector; Average distance d avg3 Exceeds the preset detection threshold d of the measurement area threshold2 , it is determined to be passable, otherwise it is impassable.

4. The method for region segmentation and autonomous target search based on environmental semantic information according to claim 1, characterized in that: The step (2) is specifically: (2.1) Use the depth camera on the robot to obtain RGB images and depth images of different objects in each area; (2.2) Using a deep learning model to obtain the center point of the object in the RGB image, and extracting the object name based on the home environment dataset; (2.3) obtaining the category of the region to which the object belongs according to the name of the object; (2.4) Acquire a depth value of the object by combining the center point information of the object with the depth image; (2.5) Obtain the robot coordinates based on the radar information and combine it with the object depth value to obtain the object coordinates; (2.6) Save the location coordinates, object names and categories of the areas to which all objects in each area belong.

5. The method for region segmentation and autonomous target search based on environmental semantic information according to claim 4, characterized in that: The step (2.5) is specifically: (2.5.1) calculating the pixel position of the robot in the map; Where: (x map ,y map ) is the current position of the robot, (x origin ,y origin ) is the position corresponding to the origin of the map, obtained according to the radar data; (u, v) is the pixel position of the robot in the map, and Resolution is the map resolution; (2.5.2) According to the pixel position of the robot in the map, the center point information of the object and the depth value of the object, the coordinates of the object in the map are obtained; Z c =d Where: (c x 、c y ) is the center point information of the object; d is the depth value of the object; (X c , Y c , Z c ) is the coordinate of the object in the map; f x 、f y is the focal length of the camera; (2.5.2) Convert to the world coordinate system to obtain the position coordinates of the object; Where: P w is the coordinate of the object; R is the rotation matrix, and t is the translation vector, which are obtained by jointly calibrating the camera and radar.

6. The method for region segmentation and autonomous target search based on environmental semantic information according to claim 4, characterized in that: The home environment data set is pre-constructed according to the names of all objects existing in the home environment.

7. The method for region segmentation and autonomous target search based on environmental semantic information according to claim 4, characterized in that: The categories of areas include living room, bedroom, kitchen, bathroom and study.

8. The method for region segmentation and autonomous target search based on environmental semantic information according to claim 1, characterized in that: The target search task includes a main task and a secondary task. The specific search process is as follows: (4.1) According to the name of the object to be found in the main task, determine the area category to which the object to be found in the main task belongs; if the sub-task is not empty, determine the area category to which the object to be found in the sub-task belongs according to the name of the object to be found in the sub-task; (4.2) The robot enters the area of ​​the object to be found in the primary task according to the map to search for the object. If the object to be found in the secondary task is detected during the process, the robot dynamically switches to the secondary task and automatically resumes the primary task after completing the secondary task. (4.3) When the main task is completed, the target search task is completed.

9. A region segmentation and autonomous target search system based on environmental semantic information, characterized in that: include: A map construction module, which is used to divide the search area into regions based on environmental semantic information and construct a map; A detection module is used to detect objects in each area in real time using a deep learning model to obtain location coordinate information and physical categories of the objects; The region naming module is used to count the frequency of objects of different categories in each region and name the current region according to the object category with the highest frequency; The target search module is used to perform target search based on the map according to the name of the object to be found in the target search task and the corresponding area category to which it belongs.

Images