Obstacle recognition system, method and device and storage medium

By using stereoscopic vision sensing modules and optical prisms in mobile robots, the reflected light of scenes is refracted into two viewing images, solving the problem of low accuracy in monocular camera recognition and achieving low cost and high accuracy obstacles Three-dimensional position recognition, suitable for autonomous navigation of mobile robots.

CN119090951APending Publication Date: 2024-12-06SHENZHEN SILVER STAR INTELLIGENT TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202310636984.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

Under cost and volume limitations, the environmental recognition accuracy of monocular cameras for mobile robots is low, and it is impossible to effectively obtain the three-dimensional position information of obstacles.

Method used

Using a stereoscopic visual sensing module, the reflected light of the scene is refracted to two different directions through optical prisms. The monocular camera collects and divides it into a first viewing angle image and a second viewing angle image. The image processor recognizes the obstacles based on the two images. Three-dimensional position information.

Benefits of technology

It improves the accuracy of obstacle position recognition, has low cost and small space, and is suitable for autonomous navigation of mobile robots.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090951A_ABST
    Figure CN119090951A_ABST
Patent Text Reader

Abstract

The invention relates to the field of robots, and discloses an obstacle recognition system, method and device and a storage medium, which are used for improving the accuracy of recognizing the position of an obstacle. The obstacle recognition system comprises a stereoscopic vision sensing module and an image processor, and the stereoscopic vision sensing module is in communication connection with the image processor. Wherein the stereoscopic vision sensing module comprises an optical prism and a monocular camera, the optical prism is used for refracting reflected light of the same scene received by the mobile robot to two different directions, and the monocular camera is used for collecting light refracted by the optical prism to generate a scene image and transmitting the scene image to the image processor; and the image processor is used for segmenting the scene image to obtain a first visual angle image and a second visual angle image, and identifying three-dimensional position information of the obstacle according to the first visual angle image and the second visual angle image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of robots, and in particular to an obstacle recognition system, method, device and storage medium. Background Art

[0002] In the process of autonomous navigation of mobile robots, it is a convenient way to use the visual system to perceive the environment. Currently, there are three main options for visual systems: the first is to identify obstacles through a monocular camera, but the monocular camera cannot obtain the distance information of the obstacles; the second is to identify obstacles through a depth camera, which can make up for the defect of not being able to obtain the distance information of obstacles under the first option, but the cost is relatively high; the third is to identify obstacles through the binocular stereo vision module of two cameras, which can achieve the same effect as the second option, but the size is large and there is a synchronization time difference between the two cameras, which affects the stability of the work.

[0003] In the prior art, household mobile robots mostly use monocular cameras for environment recognition due to cost and size limitations, which has low accuracy. Summary of the invention

[0004] The main purpose of the present invention is to provide an obstacle recognition system, method, device and storage medium, which are used to solve the problem of weak environmental perception ability of monocular cameras under cost and volume constraints, and improve the accuracy of identifying obstacle locations.

[0005] A first aspect of the present invention provides an obstacle recognition system, comprising: a stereoscopic vision sensor module and an image processor; the stereoscopic vision sensor module and the image processor are communicatively connected; wherein the stereoscopic vision sensor module comprises an optical prism and a monocular camera, the optical prism is used to refract the same scene reflected light received by the mobile robot into two different directions, the monocular camera is used to collect the light refracted by the optical prism to generate a scene image, and transmit the scene image to the image processor; the image processor is used to segment the scene image to obtain a first perspective image and a second perspective image, and identify the three-dimensional position information of the obstacle based on the first perspective image and the second perspective image.

[0006] In a feasible embodiment, the optical prism includes an optical bottom surface and two optical refractive surfaces, the two optical refractive surfaces include a first refractive surface and a second refractive surface, the optical bottom surface is used to receive the scene reflected light, and the two optical refractive surfaces are used to refract the same light from the optical bottom surface into two different directions.

[0007] In a feasible implementation manner, the optical prism is an isosceles triangle prism, the optical bottom surface is perpendicular to the optical axis of the monocular camera, and the top edge of the optical prism is perpendicular to the optical axis of the monocular camera.

[0008] In a feasible implementation, the obstacle recognition system further includes: a controller, wherein the controller is used to control the moving direction of the mobile robot to adjust the scene reflected light collected by the mobile robot.

[0009] A second aspect of the present invention provides an obstacle recognition method, which is applied to a mobile robot. The mobile robot is equipped with an obstacle recognition system, the obstacle recognition system includes a stereoscopic vision sensing module and an image processor, the stereoscopic vision sensing module includes an optical prism and a monocular camera, and the obstacle recognition method includes: photographing a target scene by the stereoscopic vision sensing module during movement to obtain a target scene image, wherein the target scene image is generated by the monocular camera collecting light refracted by the optical prism; processing the target scene image by the image processor to generate a first perspective image and a second perspective image; determining the three-dimensional position information of the target obstacle in the robot coordinate system according to the first perspective image and the second perspective image, wherein the target obstacle exists in both the first perspective image and the second perspective image.

[0010] In a feasible implementation, the target scene image is processed by the image processor to generate a first perspective image and a second perspective image, including: segmenting the target scene image to obtain a first image and a second image; acquiring a first calibration parameter corresponding to the first image and a second calibration parameter corresponding to the second image; correcting the first image according to the first calibration parameter to generate a first perspective image; and correcting the second image according to the second calibration parameter to generate a second perspective image.

[0011] In a feasible implementation, the target scene image is segmented to obtain a first image and a second image, including: identifying feature pairs having the same features on the target scene image; selecting a feature pair with the largest matching value, connecting the pixel points corresponding to the feature pair to obtain a feature line; obtaining the median perpendicular plane of the feature line, the median perpendicular plane being a plane intersecting the midpoint of the feature line and perpendicular to the feature line; and segmenting the target scene image into two images through the median perpendicular plane to obtain a first image and a second image.

[0012] In a feasible implementation, determining the three-dimensional position information of the target obstacle in the robot coordinate system based on the first perspective image and the second perspective image includes: setting a first matching area on the first perspective image, and setting a second matching area on the second perspective image, wherein the first matching area and the second matching area both contain the target obstacle; performing feature extraction and matching on the target obstacle in the first matching area and the second matching area to obtain disparity information; and determining the three-dimensional position information of the target obstacle in the robot coordinate system based on the disparity information.

[0013] In a feasible implementation manner, after determining the three-dimensional position information of the target obstacle in the robot coordinate system according to the first perspective image and the second perspective image, it also includes: mapping the three-dimensional position information of the target obstacle to a preset environment map.

[0014] In a feasible implementation, mapping the three-dimensional position information of the target obstacle to a preset environment map includes: identifying the category of the target obstacle and determining a mapping level according to the category of the target obstacle; performing coordinate conversion on the three-dimensional position information of the target obstacle to obtain the three-dimensional coordinates of the target obstacle in a map coordinate system; marking the target obstacle in the preset environment map according to the three-dimensional coordinates of the target obstacle in the map coordinate system; and adding the mapping level to the target obstacle in the preset environment map.

[0015] In a feasible implementation, the identifying the category of the target obstacle and determining the mapping level according to the category of the target obstacle include: calling a preset recognition model to identify the category of the target obstacle; when the target obstacle is a first category obstacle, determining that the mapping level of the target obstacle is a first mapping level, and the first category obstacle is an obstacle that cannot be pushed by the mobile robot; when the target obstacle is a second category obstacle, determining that the mapping level of the target obstacle is a second mapping level, and the second category obstacle is an obstacle that can be pushed by the mobile robot.

[0016] In a feasible implementation, after mapping the three-dimensional position information of the target obstacle into a preset environment map, the method further includes: obtaining obstacle information in the direction of travel from the preset environment map to obtain an obstacle set, wherein the obstacle information includes the three-dimensional coordinates and mapping level of the obstacle in the map coordinate system; judging whether there is an obstacle of the second mapping level in the obstacle set, and there is an area between the obstacle of the second mapping level and the obstacle of the first mapping level or between two obstacles of the second mapping level that cannot be cleared due to distance restrictions; if so, deleting the obstacle information of the second mapping level in the obstacle set, and planning an obstacle avoidance navigation path according to the remaining obstacle information in the obstacle set; if not, planning an obstacle avoidance navigation path according to each obstacle information in the obstacle set.

[0017] In a feasible implementation manner, before the mobile robot shoots the target scene during movement and generates the first perspective image and the second perspective image, it also includes: building an initial model of a stereoscopic vision sensor, the stereoscopic vision sensor including a monocular camera and a prism, and the prism is arranged directly in front of the monocular camera; determining the distance between the prism and the monocular camera through a target common area to obtain a final model of the stereoscopic vision sensor, wherein the target common area is an area that can be covered by both the first perspective image and the second perspective image; calibrating a preset checkerboard image through the final model of the stereoscopic vision sensor to obtain a first calibration parameter and a second calibration parameter, wherein the first calibration parameter is used to correct the first perspective image, and the second calibration parameter is used to correct the second perspective image.

[0018] The third aspect of the present invention provides an obstacle recognition device, comprising: a shooting module, used to shoot the target scene through the stereoscopic vision sensing module during movement to obtain a target scene image, wherein the target scene image is generated by the monocular camera collecting the light refracted by the optical prism; a generating module, used to process the target scene image through the image processor to generate a first perspective image and a second perspective image; and a determining module, used to determine the three-dimensional position information of the target obstacle in the robot coordinate system according to the first perspective image and the second perspective image, wherein the target obstacle exists in both the first perspective image and the second perspective image.

[0019] In a feasible implementation, the generation module includes: a segmentation unit, used to segment the target scene image to obtain a first image and a second image; an acquisition unit, used to acquire a first calibration parameter corresponding to the first image and a second calibration parameter corresponding to the second image; a first correction unit, used to correct the first image according to the first calibration parameter to generate a first perspective image; and a second correction unit, used to correct the second image according to the second calibration parameter to generate a second perspective image.

[0020] In a feasible implementation, the segmentation unit is specifically used to: identify feature pairs with the same features on the target scene image; select a feature pair with the largest matching value, connect the pixel points corresponding to the feature pair to obtain a feature line; obtain the median perpendicular plane of the feature line, the median perpendicular plane being a plane that intersects the midpoint of the feature line and is perpendicular to the feature line; and segment the target scene image into two images through the median perpendicular plane to obtain a first image and a second image.

[0021] In a feasible implementation, the determination module is specifically used to: set a first matching area on the first perspective image, and set a second matching area on the second perspective image, wherein both the first matching area and the second matching area contain target obstacles; perform feature extraction and matching on the target obstacles in the first matching area and the second matching area to obtain disparity information; and determine the three-dimensional position information of the target obstacle in the robot coordinate system according to the disparity information.

[0022] In a feasible implementation manner, the obstacle identification device further includes a mapping module, which is used to map the three-dimensional position information of the target obstacle into a preset environment map.

[0023] In a feasible implementation, the mapping module includes: a determination unit, used to identify the category of the target obstacle and determine the mapping level according to the category of the target obstacle; a conversion unit, used to perform coordinate conversion on the three-dimensional position information of the target obstacle to obtain the three-dimensional coordinates of the target obstacle in the map coordinate system; a marking unit, used to mark the target obstacle in the preset environment map according to the three-dimensional coordinates of the target obstacle in the map coordinate system; and an adding unit, used to add the mapping level for the target obstacle in the preset environment map.

[0024] In a feasible implementation, the determination unit is specifically used to: call a preset recognition model to identify the category of the target obstacle; when the target obstacle is a first category obstacle, determine that the mapping level of the target obstacle is a first mapping level, and the first category obstacle is an obstacle that cannot be pushed by the mobile robot; when the target obstacle is a second category obstacle, determine that the mapping level of the target obstacle is a second mapping level, and the second category obstacle is an obstacle that can be pushed by the mobile robot.

[0025] In a feasible implementation, the obstacle identification device also includes a planning module, which is specifically used to: obtain obstacle information in the direction of travel from the preset environment map to obtain an obstacle set, wherein the obstacle information includes the three-dimensional coordinates and mapping level of the obstacle in the map coordinate system; determine whether there is an obstacle of the second mapping level in the obstacle set, and there is an area between the obstacle of the second mapping level and the obstacle of the first mapping level or between two obstacles of the second mapping level that cannot be cleared due to distance restrictions; if so, delete the obstacle information of the second mapping level in the obstacle set, and plan an obstacle avoidance navigation path according to the remaining obstacle information in the obstacle set; if not, plan an obstacle avoidance navigation path according to each obstacle information in the obstacle set.

[0026] In a feasible implementation, the obstacle recognition device also includes a preprocessing module, which is specifically used to: build an initial model of a stereoscopic vision sensor, the stereoscopic vision sensor includes a monocular camera and a prism, and the prism is arranged directly in front of the monocular camera; determine the distance between the prism and the monocular camera through a target common area to obtain a final model of the stereoscopic vision sensor, and the target common area is an area that can be covered by both the first perspective image and the second perspective image; calibrate a preset checkerboard image through the final model of the stereoscopic vision sensor to obtain a first calibration parameter and a second calibration parameter, the first calibration parameter is used to correct the first perspective image, and the second calibration parameter is used to correct the second perspective image.

[0027] A fourth aspect of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the above-mentioned obstacle identification method.

[0028] In the technical solution provided by the present invention, an obstacle recognition system is installed in a mobile robot, and the obstacle recognition system includes a stereoscopic vision sensor module and an image processor, and the stereoscopic vision sensor module and the image processor are connected in communication, wherein the stereoscopic vision sensor module includes an optical prism and a monocular camera, and the optical prism is used to refract the same scene reflected light received by the mobile robot to two different directions, and the monocular camera is used to collect the light refracted by the prism to generate a scene image, and transmit the scene image to the image processor, and the image processor is used to segment the scene image to obtain a first perspective image and a second perspective image, and identify the three-dimensional position information of the obstacle according to the first perspective image and the second perspective image. In the embodiment of the present invention, the scene reflected light is refracted to two different directions by an optical prism, and a scene image with two perspectives is generated on the imaging plane of the monocular camera, and the scene image is segmented into a first perspective image and a second perspective image. According to the first perspective image and the second perspective image, the three-dimensional position information of the obstacle that exists in both images can be calculated, and the three-dimensional position information of the obstacle detected by the prism is added on the basis of the monocular camera, which has low cost and small space occupation, and improves the accuracy of identifying the position of the obstacle. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of an obstacle recognition system according to an embodiment of the present invention;

[0030] Figure 2 is a schematic diagram of a prism in an embodiment of the present invention;

[0031] Figure 3 A schematic diagram of an embodiment of an obstacle identification method in an embodiment of the present invention;

[0032] Figure 4 Schematic diagram of another embodiment of the obstacle identification method in an embodiment of the present invention;

[0033] Figure 5 Schematic diagram of an obstacle recognition device according to an embodiment of the present invention;

[0034] Figure 6 FIG. 2 is a schematic diagram of another embodiment of an obstacle identification device in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] Embodiments of the present invention provide an obstacle recognition system, method, device and storage medium for improving the accuracy of identifying the position of an obstacle.

[0036] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] See also Figure 1 An embodiment of an obstacle recognition system in an embodiment of the present invention is installed on a mobile robot. The obstacle recognition system includes a stereoscopic vision sensor module 110 and an image processor 120. The stereoscopic vision sensor module 110 and the image processor 120 are communicatively connected, wherein the stereoscopic vision sensor module 110 includes an optical prism 111 and a monocular camera 112. The optical prism 110 is used to refract the same scene reflected light received by the mobile robot into two different directions. The monocular camera 120 is used to collect the light refracted by the optical prism to generate a scene image, and transmit the scene image to the image processor 120. The image processor 120 is used to segment the scene image to obtain a first-perspective image and a second-perspective image, and recognize the three-dimensional position information of the obstacle according to the first-perspective image and the second-perspective image.

[0038] Optionally, the optical prism is a triangular prism, such as Figure 2 As shown, it includes an optical bottom surface 201 and two optical refractive surfaces, the two optical refractive surfaces include a first refractive surface 202 and a second refractive surface 203, the optical bottom surface 202 is used to receive scene reflected light, and the two optical refractive surfaces are used to refract the same light from the optical bottom surface into two different directions.

[0039] Furthermore, the optical prism is an isosceles prism, the optical bottom surface 201 is perpendicular to the optical axis of the monocular camera, the top edge of the optical prism is perpendicular to the optical axis of the monocular camera, and the top edge of the optical prism is the intersection edge of the first refractive surface and the second refractive surface.

[0040] Optionally, the obstacle recognition system also includes a controller 130, which is communicatively connected to the image processor 120, can receive the three-dimensional position information of the obstacle generated by the image processor 120, and control the movement of the mobile robot according to the instructions and the three-dimensional position information of the obstacle, so as to realize autonomous navigation and intelligent operation of the mobile robot.

[0041] In the embodiment of the present invention, the scene reflected light is refracted to two different directions through an optical prism, and a scene image with two perspectives is generated on the imaging plane of the monocular camera. The scene image is divided into a first perspective image and a second perspective image. The three-dimensional position information of obstacles that exist in both images at the same time can be calculated based on the first perspective image and the second perspective image. On the basis of the monocular camera, a prism is added to detect the three-dimensional position information of the obstacle, which has low cost, small space occupation, and improves the accuracy of identifying the obstacle position.

[0042] See also Figure 3 , an embodiment of the obstacle identification method in the embodiment of the present invention includes:

[0043] It is understandable that the execution subject of the present invention may be an obstacle recognition device or a mobile robot, which is not limited here. The embodiment of the present invention is described by taking a mobile robot as the execution subject.

[0044] 301. During the movement, the target scene is photographed by a stereoscopic vision sensor module to obtain a target scene image. The target scene image is generated by collecting light refracted by an optical prism through a monocular camera.

[0045] During the movement, the mobile robot collects light reflected from the surrounding environment through a stereoscopic vision sensor module at a preset frequency to generate an image. The environment collected each time the mobile robot takes a picture is a scene, and the currently photographed environment is determined as the target scene.

[0046] The mobile robot shoots the target scene through a stereoscopic vision sensing module, and the stereoscopic vision sensing module includes an optical prism and a monocular camera, wherein the optical prism includes an optical bottom surface and two optical refractive surfaces, and the two optical refractive surfaces include a first refractive surface and a second refractive surface, and the optical bottom surface is used to receive the reflected light of the target scene, and the two optical refractive surfaces are used to refract the same light from the optical bottom surface to two different directions. Preferably, the optical prism is an isosceles prism, the optical bottom surface is perpendicular to the optical axis of the monocular camera, the top edge of the optical prism is perpendicular to the optical axis of the monocular camera, and the top edge of the optical prism is the intersection edge of the first refractive surface and the second refractive surface. The reflected light of the target scene passes through the optical bottom surface and is refracted to different directions via the first refractive surface and the second refractive surface to generate a target scene image. Since the light refracted by the first refractive surface and the second refractive surface is partially the same, there are repeated object images in the target scene image.

[0047] Furthermore, after the mobile robot obtains the target scene image, it starts a deduplication program and compares the currently generated target scene image with the last generated scene image. When the similarity is higher than a preset value, it is considered that the target scene image is the same as the last generated scene image. At this time, the target scene image is deleted to avoid repeated processing of the image and improve efficiency.

[0048] 302. Process the target scene image by an image processor to generate a first-perspective image and a second-perspective image.

[0049] The mobile robot identifies obstacles appearing at two positions in the target scene graph, selects the same feature points from the obstacles at the two positions for connection, takes the midpoint of the line connecting the two feature points, and uses a plane passing through the midpoint and perpendicular to the target scene graph to split the target scene graph to obtain a first-perspective image and a second-perspective image.

[0050] 303. Determine three-dimensional position information of a target obstacle in a robot coordinate system according to the first-view image and the second-view image, where the target obstacle exists in both the first-view image and the second-view image.

[0051] The first-perspective image and the second-perspective image are equivalent to the results of two virtual cameras shooting the target scene separately at two perspectives. Therefore, there is parallax between the first-perspective image and the second-perspective image. The calibration parameters of the two virtual cameras are obtained in advance, and the mobile robot corrects the first-perspective image and the second-perspective image, and stereo matches the corrected first-perspective image and the second-perspective image to obtain parallax information. The depth information is calculated according to the parallax information, the distance between the optical prism and the monocular camera, the optical prism parameters and the monocular camera parameters to obtain the three-dimensional position information of the target obstacle.

[0052] In the embodiment of the present invention, the scene reflected light is refracted to two different directions through an optical prism, and a scene image with two perspectives is generated on the imaging plane of the monocular camera. The scene image is divided into a first perspective image and a second perspective image, and disparity information is calculated based on the first perspective image and the second perspective image, and then the depth information is calculated to obtain the three-dimensional position information of the target obstacle. The cost is low, the space occupied is small, and the accuracy of identifying the obstacle position is improved.

[0053] See also Figure 4 Another embodiment of the obstacle identification method in the embodiment of the present invention includes:

[0054] 401. During the movement, the target scene is photographed by a stereoscopic vision sensor module to obtain a target scene image. The target scene image is generated by collecting light refracted by an optical prism through a monocular camera.

[0055] In this embodiment, step 401 is the same as step 301 and will not be described again.

[0056] 402. Identify feature pairs having the same features on the target scene image.

[0057] The mobile robot identifies the features of each obstacle in the target scene image. The features of the obstacle include pixel value, shape, position, size, etc. There are multiple feature pairs on an obstacle. The features of each obstacle are matched to obtain multiple feature pairs.

[0058] The feature pairs are used to detect whether the same obstacle appears repeatedly in the target scene image. There may be multiple repeated obstacles in the target scene image, and one of the obstacles is determined as the target obstacle.

[0059] 403. Select a feature pair with the largest matching value, connect the pixel points corresponding to the feature pair, and obtain a feature connection line.

[0060] In case of inaccurate feature pair matching due to chromatic aberration caused by light angle or calculation error, the similarity of each feature pair is calculated to obtain the matching value corresponding to each feature pair. The higher the similarity, the greater the matching value. The mobile robot selects the feature pair with the largest matching value, connects the pixel points corresponding to the feature pair, and obtains the feature connection line.

[0061] Since there are repeated object images in the target scene image, the target scene image has a certain degree of symmetry. By finding the symmetrical points, the symmetrical dividing line can be obtained. Selecting a feature pair with the largest matching value is conducive to reducing errors and obtaining more accurate information.

[0062] 404. Obtain a perpendicular median plane of the feature line, where the perpendicular median plane is a plane that intersects the midpoint of the feature line and is perpendicular to the feature line.

[0063] 405. Segment the target scene image into two images using a median vertical plane to obtain a first image and a second image.

[0064] Steps 402-405 of this embodiment are a method of segmenting the target scene image into a first image and a second image. Another method of segmenting the target scene image into a first image and a second image is also provided herein, which is specifically as follows:

[0065] Calculate the median value of the number of pixel columns in the target scene graph, take the image corresponding to the pixels in the column whose number is smaller than the median value as the first-perspective image, and take the image corresponding to the pixels in the column whose number is larger than the median value as the second-perspective image, that is, divide the target scene graph in half in the column direction to obtain the first-perspective image and the second-perspective image.

[0066] 406. Obtain a first calibration parameter corresponding to the first image and a second calibration parameter corresponding to the second image, perform correction processing on the first image according to the first calibration parameter to generate a first-viewing angle image, and perform correction processing on the second image according to the second calibration parameter to generate a second-viewing angle image.

[0067] The mobile robot obtains a first calibration parameter and a second calibration parameter, which are obtained by a preprocessing process. The preprocessing process includes: building an initial model of a stereo vision sensor, the stereo vision sensor includes a monocular camera and a prism, the prism is arranged in front of the monocular camera, the distance between the prism and the monocular camera is determined by a target common area, the target common area is a scene range selected according to requirements, and a final model of the stereo vision sensor is obtained, the target common area is an area that can be covered by both the first perspective image and the second perspective image, and a preset chessboard image is calibrated by the final model of the stereo vision sensor to obtain the first calibration parameter and the second calibration parameter, the first calibration parameter is used to correct the first perspective image, and the second calibration parameter is used to correct the second perspective image.

[0068] 407. Determine three-dimensional position information of a target obstacle in the robot coordinate system according to the first-view image and the second-view image, where the target obstacle exists in both the first-view image and the second-view image.

[0069] The mobile robot sets a first matching area on the first perspective image and sets a second matching area on the second perspective image. Both the first matching area and the second matching area contain target obstacles. Feature extraction and matching are performed on the target obstacles in the first matching area and the second matching area, thereby reducing information processing that is completely irrelevant to the target obstacles, improving the matching speed, obtaining disparity information, and determining the three-dimensional position information of the target obstacle in the robot coordinate system according to the disparity information.

[0070] Furthermore, the mobile robot identifies a first outline of a target obstacle on a first-perspective image, and sets an area within the first outline as a first matching area; identifies a second outline of a target obstacle on a second-perspective image, and sets an area within the second outline as a second matching area, and performs feature extraction and matching on the target obstacles in the first matching area and the second matching area, thereby further reducing the amount of matching information and improving the matching speed.

[0071] 408. Map the three-dimensional position information of the target obstacle into a preset environment map.

[0072] The mobile robot identifies the category of the target obstacle, determines the mapping level according to the category of the target obstacle, performs coordinate conversion on the three-dimensional position information of the target obstacle to obtain the three-dimensional coordinates of the target obstacle in the map coordinate system, marks the target obstacle in a preset environment map according to the three-dimensional coordinates of the target obstacle in the map coordinate system, and adds a mapping level to the target obstacle in the preset environment map.

[0073] Specifically, the mobile robot calls a preset recognition model to identify the category of the target obstacle. When the target obstacle is a first category obstacle, the mapping level of the target obstacle is determined to be the first mapping level, and the first category obstacle is an obstacle that cannot be pushed by the mobile robot. When the target obstacle is a second category obstacle, the mapping level of the target obstacle is determined to be the second mapping level, and the second category obstacle is an obstacle that can be pushed by the mobile robot.

[0074] Furthermore, the mobile robot obtains obstacle information in the direction of travel from the preset environment map to obtain an obstacle set, wherein the obstacle information includes the three-dimensional coordinates and mapping level of the obstacle in the map coordinate system, and determines whether there is an obstacle of the second mapping level in the obstacle set, and whether there is an area between the obstacle of the second mapping level and the obstacle of the first mapping level or between two obstacles of the second mapping level that cannot be cleared due to distance restrictions. If so, the obstacle information of the second mapping level in the obstacle set is deleted, and an obstacle avoidance navigation path is planned according to the remaining obstacle information in the obstacle set. If not, an obstacle avoidance navigation path is planned according to each obstacle information in the obstacle set. After the mobile robot avoids obstacles according to the obstacle avoidance navigation path, the obstacles of the second mapping level are deleted from the preset environment map.

[0075] For example, there are three obstacles in the obstacle set obtained by the mobile robot, which are a box, slippers and a coffee table according to the distance from the mobile robot. The box and the coffee table are first-category obstacles, and the slippers are second-category obstacles. The distance 1 between the box and the slippers is calculated based on the three-dimensional coordinates of the box and the slippers, and it is found that distance 1 is not enough to allow the mobile robot to pass. The distance 2 between the coffee table and the slippers is calculated based on the three-dimensional coordinates of the coffee table and the slippers, and it is found that distance 2 is also not enough to allow the mobile robot to pass. Therefore, the slippers information in the obstacle set is deleted, and the obstacle avoidance navigation path is continued to be planned based on the remaining information of the box and the coffee table.

[0076] In the embodiment of the present invention, scene reflected light is refracted to two different directions through an optical prism, and a target scene image with two viewing angles is generated on the imaging plane of the monocular camera. Target obstacles that repeatedly appear in the target scene image are identified based on feature matching, and then the scene image is segmented into a first viewing angle image and a second viewing angle image both containing the target obstacle. Disparity information is calculated based on the first viewing angle image and the second viewing angle image, and then depth information is calculated to obtain three-dimensional position information of the target obstacle. The cost is low, the space occupied is small, the accuracy of identifying the obstacle position is improved, and the accurate obstacle position is used for navigation and positioning, thereby improving work efficiency.

[0077] The obstacle recognition method in the embodiment of the present invention is described above. The obstacle recognition device in the embodiment of the present invention is described below. Figure 5, an embodiment of the obstacle identification device in the embodiment of the present invention includes:

[0078] The shooting module 501 is used to shoot the target scene through the stereoscopic vision sensor module during the movement to obtain the target scene image, which is generated by the monocular camera collecting the light refracted by the optical prism;

[0079] A generating module 502 is used to process the target scene image through an image processor to generate a first-view image and a second-view image;

[0080] The determination module 503 is used to determine the three-dimensional position information of the target obstacle in the robot coordinate system according to the first perspective image and the second perspective image, and the target obstacle exists in both the first perspective image and the second perspective image.

[0081] In the embodiment of the present invention, the scene reflected light is refracted to two different directions through an optical prism, and a scene image with two perspectives is generated on the imaging plane of the monocular camera. The scene image is divided into a first perspective image and a second perspective image, and disparity information is calculated based on the first perspective image and the second perspective image, and then the depth information is calculated to obtain the three-dimensional position information of the target obstacle. The cost is low, the space occupied is small, and the accuracy of identifying the obstacle position is improved.

[0082] See also Figure 6 Another embodiment of the obstacle identification device in the embodiment of the present invention includes:

[0083] The shooting module 501 is used to shoot the target scene through the stereoscopic vision sensor module during the movement to obtain the target scene image, which is generated by the monocular camera collecting the light refracted by the optical prism;

[0084] A generating module 502 is used to process the target scene image through an image processor to generate a first-view image and a second-view image;

[0085] The determination module 503 is used to determine the three-dimensional position information of the target obstacle in the robot coordinate system according to the first perspective image and the second perspective image, and the target obstacle exists in both the first perspective image and the second perspective image.

[0086] Optionally, the generating module 502 includes:

[0087] The segmentation unit 5021 is used to segment the target scene image to obtain a first image and a second image;

[0088] An acquisition unit 5022, configured to acquire a first calibration parameter corresponding to the first image and a second calibration parameter corresponding to the second image;

[0089] A first correction unit 5023, configured to perform correction processing on the first image according to a first calibration parameter to generate a first viewing angle image;

[0090] The second correction unit 5024 is used to perform correction processing on the second image according to the second calibration parameters to generate a second viewing angle image.

[0091] Optionally, the segmentation unit 5021 is specifically used for:

[0092] Identify feature pairs with the same features on the target scene image; select a feature pair with the largest matching value, connect the pixel points corresponding to the feature pair to obtain the feature line; obtain the median perpendicular plane of the feature line, which is a plane that intersects the midpoint of the feature line and is perpendicular to the feature line; divide the target scene image into two images through the median perpendicular plane to obtain a first image and a second image.

[0093] Optionally, the determination module 503 is specifically configured to:

[0094] A first matching area is set on the first perspective image, and a second matching area is set on the second perspective image, wherein both the first matching area and the second matching area contain target obstacles; features of the target obstacles in the first matching area and the second matching area are extracted and matched to obtain disparity information; and three-dimensional position information of the target obstacle in the robot coordinate system is determined according to the disparity information.

[0095] Optionally, the obstacle identification device further includes a mapping module 504, which is used to map the three-dimensional position information of the target obstacle into a preset environment map.

[0096] Optionally, the mapping module 504 includes:

[0097] A determination unit 5041, configured to identify the category of the target obstacle and determine a mapping level according to the category of the target obstacle;

[0098] The conversion unit 5042 is used to perform coordinate conversion on the three-dimensional position information of the target obstacle to obtain the three-dimensional coordinates of the target obstacle in a map coordinate system;

[0099] A marking unit 5043, configured to mark the target obstacle in the preset environment map according to the three-dimensional coordinates of the target obstacle in the map coordinate system;

[0100] The adding unit 5044 is used to add the mapping level for the target obstacle in the preset environment map.

[0101] Optionally, the determining unit 5041 is specifically configured to:

[0102] A preset recognition model is called to identify the category of the target obstacle; when the target obstacle is a first category obstacle, the mapping level of the target obstacle is determined to be the first mapping level, and the first category obstacle is an obstacle that cannot be pushed by the mobile robot; when the target obstacle is a second category obstacle, the mapping level of the target obstacle is determined to be the second mapping level, and the second category obstacle is an obstacle that can be pushed by the mobile robot.

[0103] Optionally, the obstacle identification device further includes a planning module 505, and the planning module 505 is specifically used for:

[0104] Obstacle information in the direction of travel is obtained from a preset environment map to obtain an obstacle set, where the obstacle information includes the three-dimensional coordinates and mapping level of the obstacle in the map coordinate system; it is determined whether there is an obstacle of the second mapping level in the obstacle set, and there is an area between the obstacle of the second mapping level and the obstacle of the first mapping level or between two obstacles of the second mapping level that cannot be cleared due to distance restrictions; if so, the obstacle information of the second mapping level in the obstacle set is deleted, and an obstacle avoidance navigation path is planned according to the remaining obstacle information in the obstacle set; if not, an obstacle avoidance navigation path is planned according to each obstacle information in the obstacle set.

[0105] Optionally, the obstacle recognition device further includes a preprocessing module 506, and the preprocessing module 506 is specifically used for:

[0106] An initial model of a stereo vision sensor is built. The stereo vision sensor includes a monocular camera and a prism. The prism is arranged directly in front of the monocular camera. The distance between the prism and the monocular camera is determined by a target common area to obtain a final model of the stereo vision sensor. The target common area is an area covered by both the first-perspective image and the second-perspective image. The preset checkerboard image is calibrated by the final model of the stereo vision sensor to obtain a first calibration parameter and a second calibration parameter. The first calibration parameter is used to correct the first-perspective image, and the second calibration parameter is used to correct the second-perspective image.

[0107] In the embodiment of the present invention, scene reflected light is refracted to two different directions through an optical prism, and a target scene image with two viewing angles is generated on the imaging plane of the monocular camera. Target obstacles that repeatedly appear in the target scene image are identified based on feature matching, and then the scene image is segmented into a first viewing angle image and a second viewing angle image both containing the target obstacle. Disparity information is calculated based on the first viewing angle image and the second viewing angle image, and then depth information is calculated to obtain three-dimensional position information of the target obstacle. The cost is low, the space occupied is small, the accuracy of identifying the obstacle position is improved, and the accurate obstacle position is used for navigation and positioning, thereby improving work efficiency.

[0108] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the obstacle identification method.

[0109] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0111] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An obstacle recognition system, applied to a mobile robot, characterized in that: include: Stereo vision sensor module and image processor; The stereoscopic vision sensor module is communicatively connected with the image processor; The stereoscopic vision sensing module includes an optical prism and a monocular camera, wherein the optical prism is used to refract the reflected light of the same scene received by the mobile robot into two different directions, and the monocular camera is used to collect the light refracted by the optical prism to generate a scene image, and transmit the scene image to the image processor; The image processor is used to segment the scene image to obtain a first-view image and a second-view image, and recognize three-dimensional position information of an obstacle according to the first-view image and the second-view image.

2. The obstacle recognition system according to claim 1, characterized in that: The optical prism includes an optical bottom surface and two optical refractive surfaces, the two optical refractive surfaces include a first refractive surface and a second refractive surface, the optical bottom surface is used to receive the scene reflected light, and the two optical refractive surfaces are used to refract the same light from the optical bottom surface into two different directions.

3. The obstacle recognition system according to claim 2, characterized in that: The optical prism is an isosceles triangle prism, the optical bottom surface is perpendicular to the optical axis of the monocular camera, and the top edge of the optical prism is perpendicular to the optical axis of the monocular camera.

4. The obstacle recognition system according to claim 1, characterized in that: Also includes: A controller is used to control the moving direction of the mobile robot to adjust the scene reflected light collected by the mobile robot.

5. An obstacle recognition method, applied to a mobile robot, wherein the mobile robot is equipped with an obstacle recognition system, wherein the obstacle recognition system comprises a stereoscopic vision sensor module and an image processor, wherein the stereoscopic vision sensor module comprises an optical prism and a monocular camera, wherein: The obstacle recognition method comprises: During the movement, the target scene is photographed by the stereoscopic vision sensing module to obtain a target scene image, wherein the target scene image is generated by the monocular camera collecting the light refracted by the optical prism; Processing the target scene image by the image processor to generate a first perspective image and a second perspective image; The three-dimensional position information of the target obstacle in the robot coordinate system is determined according to the first perspective image and the second perspective image, and the target obstacle exists in the first perspective image and the second perspective image at the same time.

6. The obstacle recognition method according to claim 5, characterized in that: The processing of the target scene image by the image processor to generate a first-view image and a second-view image includes: Segmenting the target scene image to obtain a first image and a second image; Acquire a first calibration parameter corresponding to the first image and a second calibration parameter corresponding to the second image; Performing correction processing on the first image according to the first calibration parameter to generate a first viewing angle image; The second image is corrected according to the second calibration parameter to generate a second viewing angle image.

7. The obstacle recognition method according to claim 6, characterized in that: The step of segmenting the target scene image to obtain a first image and a second image includes: Identifying feature pairs having the same features on the target scene image; Select a feature pair with the largest matching value, connect the pixel points corresponding to the feature pair, and obtain a feature connection line; Obtaining a perpendicular midplane of the characteristic line, wherein the perpendicular midplane is a plane intersecting the midpoint of the characteristic line and being perpendicular to the characteristic line; The target scene image is divided into two images by the mid-vertical plane to obtain a first image and a second image.

8. The obstacle recognition method according to claim 5, characterized in that: The determining the three-dimensional position information of the target obstacle in the robot coordinate system according to the first perspective image and the second perspective image includes: Setting a first matching area on the first perspective image, and setting a second matching area on the second perspective image, wherein both the first matching area and the second matching area contain a target obstacle; Extracting and matching features of target obstacles in the first matching area and the second matching area to obtain disparity information; The three-dimensional position information of the target obstacle in the robot coordinate system is determined according to the parallax information.

9. The obstacle recognition method according to claim 5, characterized in that: After determining the three-dimensional position information of the target obstacle in the robot coordinate system according to the first perspective image and the second perspective image, the method further includes: The three-dimensional position information of the target obstacle is mapped into a preset environment map.

10. The obstacle recognition method according to claim 9, characterized in that: Mapping the three-dimensional position information of the target obstacle into a preset environment map includes: Identifying the category of the target obstacle, and determining a mapping level according to the category of the target obstacle; Performing coordinate conversion on the three-dimensional position information of the target obstacle to obtain the three-dimensional coordinates of the target obstacle in a map coordinate system; Marking the target obstacle in the preset environment map according to the three-dimensional coordinates of the target obstacle in the map coordinate system; The mapping level is added for the target obstacle in the preset environment map.

11. The obstacle recognition method according to claim 10, characterized in that: The identifying the category of the target obstacle and determining the mapping level according to the category of the target obstacle includes: Calling a preset recognition model to identify the category of the target obstacle; When the target obstacle is a first category obstacle, determining the mapping level of the target obstacle to be the first mapping level, and the first category obstacle is an obstacle that cannot be pushed by the mobile robot; When the target obstacle is a second category obstacle, the mapping level of the target obstacle is determined to be the second mapping level, and the second category obstacle is an obstacle that can be pushed by a mobile robot.

12. The obstacle recognition method according to claim 11, characterized in that: After mapping the three-dimensional position information of the target obstacle into a preset environment map, the method further includes: Obtaining obstacle information in the traveling direction from the preset environment map to obtain an obstacle set, wherein the obstacle information includes three-dimensional coordinates and mapping levels of the obstacles in a map coordinate system; Determine whether there is an obstacle of the second mapping level in the obstacle set, and whether there is an area that cannot be cleared due to distance limitation between an obstacle of the second mapping level and an obstacle of the first mapping level or between two obstacles of the second mapping level; If yes, deleting the obstacle information of the second mapping level in the obstacle set, and planning an obstacle avoidance navigation path according to the remaining obstacle information in the obstacle set; If not, an obstacle avoidance navigation path is planned according to the information of each obstacle in the obstacle set.

13. The obstacle recognition method according to any one of claims 5 to 12, characterized in that: Before the mobile robot shoots the target scene during movement to generate the first-view image and the second-view image, the method further includes: Building an initial model of a stereoscopic vision sensor, wherein the stereoscopic vision sensor includes a monocular camera and a prism, wherein the prism is arranged directly in front of the monocular camera; Determining the distance between the prism and the monocular camera through a target common area to obtain a final model of the stereo vision sensor, wherein the target common area is an area that can be covered by both the first perspective image and the second perspective image; The preset checkerboard image is calibrated through the final model of the stereo vision sensor to obtain a first calibration parameter and a second calibration parameter, wherein the first calibration parameter is used to correct the first perspective image, and the second calibration parameter is used to correct the second perspective image.

14. An obstacle recognition device, characterized in that: The obstacle recognition device comprises: A shooting module, used for shooting a target scene through the stereoscopic vision sensing module during movement to obtain a target scene image, wherein the target scene image is generated by the monocular camera collecting the light refracted by the optical prism; A generating module, configured to process the target scene image through the image processor to generate a first-view image and a second-view image; A determination module is used to determine the three-dimensional position information of a target obstacle in a robot coordinate system according to the first perspective image and the second perspective image, wherein the target obstacle exists in both the first perspective image and the second perspective image.

15. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the obstacle recognition method as described in any one of claims 5-13 is implemented.

Citation Information

Patent Citations

  • Monocular camera three-dimensional reconstruction method and system based on rotary refraction sheet

    CN111768486A

  • Target three-dimensional identification method and device based on single camera and medium

    CN114241028A

  • Multi-perspective video acquisition apparatus and vehicle

    JP2016031267A

  • Mobile control method, mobile robot and computer storage medium

    US20210138657A1