Obstacle detection method and device, mowing robot and storage medium

Abstract

This application discloses an obstacle detection method, apparatus, lawnmower robot, and storage medium, comprising: acquiring at least one initial image of the current scene; extracting disparity information and semantic segmentation information from the initial image; matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information; and calculating the position and size of the obstacle based on the obstacle disparity information. This application can detect the current scene using only a visual sensor and calculate the position and size of obstacles in the current scene based on disparity information and semantic segmentation information, without requiring other types of sensors, and has high accuracy.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an obstacle detection method, device, lawnmower robot, and storage medium. Background Technology

[0002] As people's living standards continue to improve, they have increasingly higher requirements for their leisure environment. Private gardens, parks, and playgrounds have become the best places for people to relax and enjoy themselves. However, the lawns in private gardens, parks, and playgrounds need to be maintained regularly to ensure their aesthetic appeal. Currently, lawn mowing robots are commonly used to replace manual mowing.

[0003] However, lawn mowing robots often encounter various obstacles during their work. Existing lawn mowing robots can detect obstacles using recognition devices installed on the robot, such as LiDAR, millimeter-wave radar, and multi-sensor information fusion methods. Although the methods mentioned above can effectively detect obstacles, they have high requirements in terms of manufacturing and maintenance costs. Summary of the Invention

[0004] This application provides an obstacle detection method, device, lawnmower robot, and storage medium, which can detect the position and size of obstacles using only a visual sensor and has high accuracy.

[0005] In a first aspect, embodiments of this application provide an obstacle detection method, including:

[0006] Obtain at least one initial image for the current scene;

[0007] Extract disparity information and semantic segmentation information from the initial image;

[0008] The semantic segmentation information is matched with the disparity information to determine the obstacle disparity information in the disparity information;

[0009] The position and size of the obstacle are calculated based on the obstacle parallax information.

[0010] In one embodiment, before matching the semantic segmentation information with the disparity information, the method further includes:

[0011] The image segmentation model is trained using disparity information samples and the corresponding label information.

[0012] The trained image segmentation model segments the disparity information in the initial image into foreground disparity information and background disparity information.

[0013] In one embodiment, matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information includes:

[0014] The semantic segmentation information is matched with the foreground disparity information to determine whether there are obstacles in the foreground;

[0015] If it exists, then obtain the obstacle disparity information from the foreground disparity information;

[0016] If it does not exist, the semantic segmentation information is matched with the background disparity information to obtain the obstacle disparity information in the background disparity information.

[0017] In one embodiment, extracting semantic segmentation information from the initial image includes:

[0018] Extract edge and depth information from the initial image;

[0019] The semantic segmentation information in the initial image is determined based on the edge information and depth information.

[0020] In one embodiment, matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information includes:

[0021] Determine whether the corresponding pixel in the semantic segmentation map for each pixel in the disparity map belongs to the semantic category of obstacle;

[0022] If so, the pixel is retained, and obstacle parallax information is determined based on all the retained pixels.

[0023] In one embodiment, calculating the position of the obstacle based on the obstacle parallax information includes:

[0024] The outline of the obstacle is determined in the initial image;

[0025] A spatial rectangular coordinate system is established based on the initial image, and the x and y values ​​of the center point of the contour are obtained;

[0026] The depth value of the center point of the contour is calculated based on the obstacle parallax information and used as the z-value.

[0027] In one embodiment, the method further includes: constructing a training set using image samples and depth labels corresponding to the image samples to train a preset convolutional neural network;

[0028] The initial image is input into the trained depth estimation network, which outputs a second depth value for the center point of the contour.

[0029] In one embodiment, after outputting a second depth value for the center point of the contour, the method further includes:

[0030] Calculate the difference between the first depth and the second depth values, and determine whether the difference is less than a preset value;

[0031] If the value is less than the specified value, the first depth value is determined to be successfully verified, and the position of the obstacle is output based on the x, y, and z values ​​of the center point of the contour.

[0032] In one embodiment, calculating the size of the obstacle based on the obstacle parallax information includes:

[0033] Calculate the coordinates of at least one edge point of the obstacle in the spatial rectangular coordinate system;

[0034] The size of the obstacle is calculated based on the outline of the obstacle and the coordinates of the at least one edge point.

[0035] Secondly, embodiments of this application provide an obstacle detection device, comprising:

[0036] The acquisition module is used to acquire at least one initial image for the current scene;

[0037] The extraction module is used to extract disparity information and semantic segmentation information from the initial image;

[0038] A matching module is used to match the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information;

[0039] The calculation module is used to calculate the position and size of the obstacle based on the obstacle parallax information.

[0040] Thirdly, embodiments of this application provide a lawnmower robot, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the obstacle detection method described above.

[0041] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the obstacle detection method described above.

[0042] The obstacle detection method provided in this application can acquire at least one initial image of the current scene, extract disparity information and semantic segmentation information from the initial image, match the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information, and calculate the position and size of the obstacle based on the obstacle disparity information. This application can detect the current scene using only a visual sensor and calculate the position and size of obstacles in the current scene based on disparity information and semantic segmentation information, without requiring other types of sensors, and has high accuracy. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a scenario for the obstacle detection method provided in the embodiments of this application;

[0045] Figure 2 This is a schematic flowchart of an obstacle detection method provided in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram of a route design provided in an embodiment of this application;

[0047] Figure 4 This is a semantic segmentation diagram provided in an embodiment of this application;

[0048] Figure 5 This is another schematic flowchart of the obstacle detection method provided in the embodiments of this application;

[0049] Figure 6 This is a schematic diagram of the obstacle detection device provided in an embodiment of this application;

[0050] Figure 7 This is another structural schematic diagram of the obstacle detection device provided in the embodiments of this application;

[0051] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0052] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be for both fixing and circuit connection purposes.

[0054] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0055] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0056] This application provides an obstacle detection method, apparatus, lawnmower robot, and storage medium.

[0057] Specifically, the obstacle detection device can be integrated into the microcontroller unit (MCU) of the lawnmower robot, or it can be integrated into a smart terminal or server. An MCU, also known as a single-chip microcomputer, is a chip-level computer formed by appropriately reducing the frequency and specifications of a central processing unit (CPU) and integrating peripheral interfaces such as memory, timer, USB, analog-to-digital converter / digital-to-analog converter, UART, PLC, and DMA, allowing for different control combinations for different applications. The lawnmower robot can walk automatically, avoid collisions, and automatically return to its charging station within its designated range. It features safety detection and battery level monitoring, and has a certain climbing ability. It is particularly suitable for lawn mowing and maintenance in home gardens and public green spaces. Its characteristics include: automatic mowing, grass clipping removal, automatic rain protection, automatic charging, automatic obstacle avoidance, compact size, electronic virtual fence, and network control.

[0058] The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and the server can be connected directly or indirectly through wired or wireless communication. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. This application does not impose any restrictions on these aspects.

[0059] For example, please see Figure 1 This application provides a lawnmowing system, including a lawnmowing robot 10, a server 20, and a user device 30, which are interconnected. The user can control the lawnmowing robot 10 to detect obstacles during its movement via the user device 30. Specifically, in this embodiment, the lawnmowing robot 10 can integrate a vision sensor to detect obstacles. In other embodiments, the lawnmowing robot 10 can also integrate other sensors, such as a rain sensor on the top to detect rain, lateral ultrasonic sensors and infrared sensors on the sides, and collision sensors around its perimeter. Multiple sensors can be used to comprehensively detect obstacles. When an obstacle is detected, the lawnmowing robot 10 can then detour around it.

[0060] During the operation, users can control and adjust the movement path, speed, and mowing range of the lawnmower robot 10 in real time via user device 30. After the operation is completed, the data corresponding to the mowing operation can be synchronized to server 20 for user viewing.

[0061] For example, the lawnmower robot 10 responds to user instructions to start working. These user instructions can be generated and sent through the user device 30. During the work process, the lawnmower robot 10 detects obstacles in the current path and generates an obstacle avoidance path based on the detection results to avoid the obstacles.

[0062] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0063] An obstacle detection method includes: acquiring at least one initial image for a current scene; extracting disparity information and semantic segmentation information from the initial image; matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information; and calculating the position and size of the obstacle based on the obstacle disparity information.

[0064] Please see Figure 2 , Figure 2 This is a schematic flowchart of the obstacle detection method provided in an embodiment of this application. The specific flow of the obstacle detection method can be as follows:

[0065] 101. Obtain at least one initial image for the current scene.

[0066] In one embodiment, the lawnmower robot can mow lawns in a pre-defined work area and plan its corresponding movement path based on that work area. For example... Figure 3 As shown, once the lawnmower starts operating, it will generate a movement path throughout the entire current area to determine if the movement path can cover the entire current area. For example, it can set a turnaround point based on the boundary of the current area and generate an arc-shaped movement path based on the turnaround point as the movement path, and then perform lawnmowing operations according to the movement path.

[0067] In one embodiment, while the lawnmower is moving along the initial path described above, it can also detect obstacles in the current path and detour around them. Specifically, this detection can be achieved using integrated visual sensors on the lawnmower, such as multiple cameras capturing images of the current scene to obtain at least one initial image. In other embodiments, the lawnmower can also integrate other types of sensors, such as collision sensors, depth sensors, ultrasonic sensors, and infrared sensors, etc.

[0068] Specifically, once the vision sensor installed on the lawnmower is turned on, it can collect an initial image in real time or at a fixed working frequency and at preset intervals, thereby continuously acquiring at least one initial image for the current scene.

[0069] 102. Extract disparity information and semantic segmentation information from the initial image.

[0070] In one embodiment, the visual sensor mounted on the lawnmower robot can be a binocular camera, including a left camera and a right camera. By simultaneously capturing images of the current scene from different perspectives using the binocular cameras, two images of the current scene can be obtained, and then the disparity information in the images can be calculated. The disparity information refers to the difference in pixel coordinates of a point in the same world coordinate system in different images, and is usually represented in the form of an image, i.e., a disparity map.

[0071] Furthermore, embodiments of this application can also perform semantic segmentation on the initial image to classify each pixel in the initial image, that is, to divide the initial image into multiple blocks according to the similarity or difference of categories, such as... Figure 4 As shown. Specifically, semantic segmentation of images can be performed using methods such as Normalized-cut, Structured Random Forest, and SVM (Support Vector Machine). In this embodiment, semantic segmentation can also be performed using deep learning.

[0072] Specifically, the semantic segmentation process described above typically includes three steps: training, validation, and testing. During training, the training samples and labels from the dataset are fed into the convolutional neural network (CNN) according to the batch size (the number of parameters used for training). Preprocessing of the training images and labels, such as cropping and data augmentation, should be performed as needed. This benefits the training of deep networks, accelerates the convergence process, avoids overfitting, and enhances the model's generalization ability. After one epoch of training, the validation samples and labels from the dataset are fed into the CNN, along with the training weights. Validation is performed using a pre-defined semantic segmentation metric to obtain the metric score for the current training process, and the corresponding weights are saved. A method of training and validating simultaneously is commonly used to better supervise model performance. After all training is complete, the test samples and labels from the dataset are fed into the CNN, and the best saved weight values ​​are loaded into the model for testing. The test results are divided into two types: one measures network performance based on commonly used metric scores, and the other saves the network's prediction results as images to intuitively assess the segmentation accuracy.

[0073] 103. Match semantic segmentation information with disparity information to determine obstacle disparity information in the disparity information.

[0074] In one embodiment, after obtaining the disparity information and semantic segmentation information corresponding to the initial image, obstacles in the initial image and their corresponding disparity information can be further determined based on the semantic segmentation results. Specifically, since the semantic segmentation information includes the category information of each object in the initial image, it is possible to determine which objects are obstacles based on the category information, and then further determine the disparity information of the obstacles by combining the disparity information.

[0075] Specifically, each pixel in the disparity map can be matched with its corresponding semantic segmentation map to determine whether the pixel belongs to the obstacle category. If it does, the pixel is retained; otherwise, it is removed. The image composed of all the retained pixels is the obstacle image, and the disparity information corresponding to the obstacle is obtained from the disparity map. In other words, the step of matching the semantic segmentation information with the disparity information to determine the obstacle disparity information can include: determining whether the corresponding pixel in the semantic segmentation map for each pixel in the disparity map belongs to the obstacle category; if so, retaining the pixel, and determining the obstacle disparity information based on all the retained pixels.

[0076] 104. Calculate the position and size of obstacles based on obstacle parallax information.

[0077] In one embodiment, after obtaining the parallax information of the obstacle, the position and size information of the obstacle can be further calculated, and the lawnmower robot can design an obstacle avoidance route to complete the lawnmower operation. Specifically, the three-dimensional coordinates of each vertex of the obstacle in the current spatial coordinate system can be calculated using the parallax information, thereby calculating the volume of the obstacle.

[0078] For example, the MCU in a lawnmower can generate an obstacle avoidance route based on the location and size of obstacles, and then control the lawnmower to perform the mowing operation based on the obstacle avoidance route; another example is that a server or user device can control the lawnmower to move according to the obstacle avoidance route, thereby performing the mowing operation. That is, the lawnmower performs the mowing operation according to the obstacle avoidance route, and after completing the obstacle avoidance route, it is controlled to continue to follow the initial movement path and continue to detect obstacles.

[0079] As can be seen from the above, the obstacle detection method provided in this application can acquire at least one initial image of the current scene, extract disparity information and semantic segmentation information from the initial image, match the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information, and calculate the position and size of the obstacle based on the obstacle disparity information. This application can detect the current scene using only a visual sensor and calculate the position and size of obstacles in the current scene based on disparity information and semantic segmentation information, without requiring other types of sensors, and has high accuracy.

[0080] Please see Figure 5 , Figure 5 Another schematic flowchart of the obstacle detection method provided in this application embodiment is shown. The specific flow of the obstacle detection method can be as follows:

[0081] 201. Obtain at least one initial image for the current scene.

[0082] In one embodiment, the visual sensor installed on the lawnmower robot can be a binocular camera, including a left camera and a right camera. By simultaneously capturing images of the current scene with the binocular camera, two images of the current scene based on different perspectives can be obtained.

[0083] 202. Extract disparity information and semantic segmentation information from the initial image.

[0084] In one embodiment, a coordinate system can be established in the current scene, and the difference in coordinates of the same point in the two images captured by the binocular camera is the disparity information. The disparity information of the initial image can be represented by a disparity map.

[0085] Furthermore, edge information and depth information of objects in the initial image can be obtained to classify objects of different categories in the image and ultimately obtain semantic segmentation information. Specifically, the edge information can be extracted from the initial image using the Canny edge detection algorithm, and the depth information can be extracted using the feature extraction layer of a VGG (visual geometry group) network. This VGG network can include 16 layers, each with a 3x3 convolutional kernel, and the number of channels in the four convolutional computation layers are 64, 128, 256, and 512, respectively. After obtaining the edge and depth information of the initial image, feature fusion can be performed to obtain a semantic segmentation map. That is, the step of extracting semantic segmentation information from the initial image can include: extracting edge and depth information from the initial image, and determining semantic segmentation information from the initial image based on the edge and depth information.

[0086] 203. The image segmentation model is trained using disparity information samples and the corresponding label information.

[0087] In one embodiment, the image segmentation model described above can be the U-Net model. It should be noted that the U-Net model does not directly pool and upsample the image to the same size as the original image. Instead, it uses deconvolution to map the high-dimensional features back to the low dimension. During the mapping process, in order to enhance the segmentation accuracy, it will fuse images with the same dimension in the shrinking network. Since the dimension will become twice the original dimension during the fusion process, it is necessary to convolve again to ensure that the dimension after processing is the same as the dimension before the fusion operation, so that it can be fused with the image with the same dimension after another deconvolution, until the final output image has the same dimension as the original image.

[0088] In the process of training the U-net model in advance, the disparity map obtained from the upstream algorithm can be used to create a training dataset. The supervision labels are self-built corresponding foreground and background images. Then, the U-net model is trained in a supervised manner using the prepared dataset and the real labels, and its output is foreground and background information.

[0089] 204. The trained image segmentation model segments the disparity information in the initial image into foreground disparity information and background disparity information.

[0090] In practical use, considering that some background objects in the scene may also be identified as obstacles, even though they do not actually interfere with the lawnmower robot, it is necessary to distinguish the current scene into foreground and background. In one embodiment, after the above-mentioned U-net model is trained, the disparity map can be further separated into foreground and background to obtain the disparity information of the foreground and background.

[0091] 205. Match the semantic segmentation information with the foreground disparity information to determine whether there are obstacles in the foreground. If yes, proceed to step 206; otherwise, proceed to step 207.

[0092] 206. Obtain obstacle parallax information from the foreground parallax information.

[0093] 207. Match the semantic segmentation information with the background disparity information to obtain the obstacle disparity information in the background disparity information.

[0094] In one embodiment, after obtaining the disparity information and semantic segmentation information corresponding to the initial image, obstacles in the initial image and their corresponding disparity information can be further determined based on the semantic segmentation results. Specifically, this embodiment prioritizes obtaining the disparity information of obstacles in the foreground. If there are no obstacles in the foreground, obstacle information in the background can be further obtained.

[0095] 208. Calculate the position and size of the obstacle based on the obstacle parallax information.

[0096] In one embodiment, the location of the obstacle can be selected from the center point of the obstacle's outline. For example, a spatial Cartesian coordinate system can be established based on the initial image. The pixel position of the center point in the image is the x, y value of the obstacle in the camera coordinate system, and its z value can be represented by a depth value. Specifically, the disparity value corresponding to the point can be converted into a depth value as the z value. Here, the x, y, and z values ​​represent the horizontal axis coordinate, vertical axis coordinate, and vertical axis coordinate in the spatial Cartesian coordinate system, respectively.

[0097] In this embodiment, the depth value of the aforementioned center point can be calculated based on the disparity information of the binocular image. Specifically, the depth value can be calculated by dividing the product of the focal length and the baseline by the disparity. The focal length is the focal length of the initial image captured by the binocular camera, i.e., the distance from the optical center of the lens to the imaging plane such as the film, CCD, or CMOS. The baseline is the distance between the optical centers of the two cameras of the binocular camera. Therefore, the step of calculating the position of the obstacle based on the obstacle disparity information can include: determining the outline of the obstacle in the initial image, establishing a spatial rectangular coordinate system based on the initial image, obtaining the x and y values ​​of the center point of the outline, and calculating the depth value of the center point of the outline based on the obstacle disparity information and using it as the z value.

[0098] In another embodiment, the depth value of the aforementioned center point can also be calculated based on a single image from either a monocular or binocular image. For example, the depth value can be output by inputting a monocular image into a trained depth estimation network, which can be a convolutional neural network. This requires preparing a large training set beforehand to train the network parameters of the convolutional neural network. Each training set can include a regular color image of an outdoor scene taken from a certain angle, along with the corresponding depth label. The depth label can be created using a binocular depth estimation algorithm. Optionally, the SYNTHIA dataset can also be used as a training set to train the convolutional neural network. This dataset is collected by simulating a car driving in real traffic conditions, taking photos from a fixed position and angle on the car at regular intervals. Therefore, the dataset will contain many sets of almost identical images. Redundant data can be removed by considering the car's speed, thereby reducing the computational load.

[0099] During the training of the convolutional neural network described above, a loss function can be used to iteratively update the network parameters to obtain a trained depth estimation network. In one embodiment, during model training, the training set is input into the network model, and predicted values ​​are obtained through the forward pass. Then, the difference between the predicted value and the true value is calculated using a loss function. The value of the loss function is then propagated back to the trained network model to optimize various parameters in the model, thereby reducing the loss value. This allows the predicted values ​​generated by the model to continuously approach the true values, thus completing the iterative update.

[0100] Optionally, when the depth value of the center point is calculated based on the disparity information of the binocular image, the depth value can be further verified. Specifically, in this embodiment, after calculating the first depth value using the disparity information of the binocular image, a second depth value can be output using a monocular image from the binocular image through the aforementioned depth estimation network. Then, the first depth value is verified based on the second depth value, for example, by comparing the difference between the first and second depth values ​​to determine whether it is greater than a preset difference, thereby determining whether there is an error in the calculation of the depth value of the center point. When the error is large, the initial image can be re-acquired and the disparity information and depth value can be calculated again.

[0101] In one embodiment, the step of calculating the size of the obstacle based on the obstacle parallax information may include: calculating the coordinates of at least one edge point of the obstacle in the spatial rectangular coordinate system, and calculating the size of the obstacle based on the outline of the obstacle and the coordinates of the at least one edge point.

[0102] As can be seen from the above, the obstacle detection method provided in this application can acquire at least one initial image of the current scene, extract disparity information and semantic segmentation information from the initial image, train an image segmentation model using disparity information samples and corresponding label information, segment the disparity information in the initial image into foreground disparity information and background disparity information using the trained image segmentation model, match the semantic segmentation information with the foreground disparity information to determine whether there is an obstacle in the foreground, if so, obtain the obstacle disparity information in the foreground disparity information, if not, match the semantic segmentation information with the background disparity information to obtain the obstacle disparity information in the background disparity information, and calculate the position and size of the obstacle based on the obstacle disparity information. This application can detect the current scene using only a visual sensor and calculate the position and size of obstacles in the current scene based on disparity information and semantic segmentation information, without requiring other types of sensors, and has high accuracy.

[0103] To facilitate better implementation of the obstacle detection method of this application embodiment, this application embodiment also provides an obstacle detection device based on the above description. The meanings of the terms used are the same as in the obstacle detection method described above, and specific implementation details can be found in the description of the method embodiment.

[0104] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of the obstacle detection device provided in the embodiments of this application, wherein the obstacle detection device may include:

[0105] The acquisition module 301 is used to acquire at least one initial image for the current scene;

[0106] Extraction module 302 is used to extract disparity information and semantic segmentation information from the initial image;

[0107] The matching module 303 is used to match the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information;

[0108] The calculation module 304 is used to calculate the position and size of the obstacle based on the obstacle parallax information.

[0109] In one embodiment, please refer to Figure 7 Extraction module 302 specifically includes:

[0110] Extraction submodule 3021 is used to extract edge information and depth information from the initial image;

[0111] The first determining submodule 3022 is used to determine semantic segmentation information in the initial image based on the edge information and depth information.

[0112] In one embodiment, the matching module 303 may include:

[0113] The judgment submodule 3031 is used to determine whether the corresponding pixel in the semantic segmentation map for each pixel in the disparity map belongs to the semantic category of obstacle;

[0114] The second determining submodule 3032 is used to retain the pixel points when the judging submodule 3031 determines that the pixel points are true, and to determine the obstacle parallax information based on all the retained pixel points.

[0115] As can be seen from the above, in this embodiment, the acquisition module 301 acquires at least one initial image of the current scene, the extraction module 302 extracts disparity information and semantic segmentation information from the initial image, the matching module 303 matches the semantic segmentation information with the disparity information to determine the obstacle disparity information, and the calculation module 304 calculates the position and size of the obstacle based on the obstacle disparity information. This application can detect the current scene using only a visual sensor and calculate the position and size of obstacles in the current scene based on disparity information and semantic segmentation information, without requiring other types of sensors, and has high accuracy.

[0116] Furthermore, embodiments of this application also provide a lawnmower robot, such as... Figure 8 As shown, it illustrates the structural diagram of the lawnmower robot involved in the embodiments of this application, specifically:

[0117] The lawnmower robot may include components such as a control module 501, a traveling mechanism 502, a cutting module 503, and a power supply 504. Those skilled in the art will understand that... Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein:

[0118] The control module 501 is the control center of the lawnmower robot. Specifically, the control module 501 may include a central processing unit (CPU), memory, input / output ports, system bus, timer / counter, digital-to-analog converter, and analog-to-digital converter, etc. The CPU executes various functions and processes data of the lawnmower robot by running or executing software programs and / or modules stored in the memory, and by calling data stored in the memory. Preferably, the CPU may integrate an application processor and a modem processor. The application processor mainly handles the operating system and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the CPU.

[0119] Memory is used to store software programs and modules. The CPU executes various functional applications and data processing by running the software programs and modules stored in memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area can store data created based on the use of the electronic device. Furthermore, memory can include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the CPU with access to the memory.

[0120] The traveling mechanism 502 is electrically connected to the control module 501 and is used to respond to the control signals transmitted by the control module 501 to adjust the traveling speed and direction of the lawn mower robot, thereby realizing the self-moving function of the lawn mower robot.

[0121] The cutting module 503 is electrically connected to the control module 501 and is used to respond to the control signals transmitted by the control module to adjust the height and speed of the cutting disc to achieve grass cutting.

[0122] Power supply 504 can be logically connected to control module 501 through power management system, thereby realizing functions such as charging, discharging, and power consumption management through power management system. Power supply 504 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0123] Although not shown, the lawnmower robot may also include a communication module, a sensor module, a prompting module, etc., which will not be described in detail here.

[0124] The communication module is used for receiving and sending signals during the information transmission and reception process. It establishes a communication connection with user equipment, base stations, or servers to realize the transmission and reception of signals with user equipment, base stations, or servers.

[0125] The sensor module is used to collect internal or external environmental information and feed the collected environmental data back to the control module for decision-making, enabling the lawnmower robot to achieve precise positioning and intelligent obstacle avoidance. Optionally, the sensors may include: ultrasonic sensors, infrared sensors, collision sensors, rain sensors, lidar sensors, inertial measurement units, wheel speed sensors, image sensors, position sensors, and other sensors, without limitation.

[0126] The notification module is used to inform the user of the current working status of the lawnmower robot. In this solution, the notification module includes, but is not limited to, indicator lights and buzzers. For example, the lawnmower robot can use indicator lights to indicate the current power status, motor operating status, and sensor operating status. As another example, when a malfunction or theft of the lawnmower robot is detected, an alarm can be triggered via a buzzer.

[0127] Specifically, in this embodiment, the processor in the control module 501 loads the executable files corresponding to the processes of one or more applications into the memory according to the following instructions, and the processor runs the applications stored in the memory to achieve various functions, as follows:

[0128] Acquire at least one initial image for the current scene, extract disparity information and semantic segmentation information from the initial image, match the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information, and calculate the position and size of the obstacle based on the obstacle disparity information.

[0129] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0130] This application embodiment can acquire at least one initial image of the current scene, extract disparity information and semantic segmentation information from the initial image, match the semantic segmentation information with the disparity information to determine obstacle disparity information, and calculate the position and size of the obstacle based on the obstacle disparity information. This application can detect the current scene using only a visual sensor and calculate the position and size of obstacles in the current scene based on disparity information and semantic segmentation information, without requiring other types of sensors, and has high accuracy.

[0131] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0132] Therefore, embodiments of this application provide a storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the obstacle detection methods provided in embodiments of this application. For example, the instructions can execute the following steps:

[0133] Acquire at least one initial image for the current scene, extract disparity information and semantic segmentation information from the initial image, match the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information, and calculate the position and size of the obstacle based on the obstacle disparity information.

[0134] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0135] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0136] Since the instructions stored in the storage medium can execute the steps of any obstacle detection method provided in the embodiments of this application, the beneficial effects that any obstacle detection method provided in the embodiments of this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0137] The foregoing has provided a detailed description of an obstacle detection method, apparatus, lawnmower robot, and storage medium provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An obstacle detection method characterized by, include: Obtain at least one initial image for the current scene; Disparity information and semantic segmentation information are extracted from the initial image. The disparity information includes foreground disparity information and background disparity information. The semantic segmentation information refers to classifying objects of different categories in the initial image. The method of extracting the semantic segmentation information includes: extracting edge information and depth information from the initial image, and performing feature fusion based on the edge information and depth information to obtain the semantic segmentation information in the initial image. Matching the semantic segmentation information with the disparity information to determine obstacle disparity information in the disparity information includes: matching the semantic segmentation information with the foreground disparity information to determine whether there is an obstacle in the foreground; If it exists, then obtain the obstacle disparity information from the foreground disparity information; If it does not exist, the semantic segmentation information is matched with the background disparity information to obtain the obstacle disparity information in the background disparity information; The position and size of the obstacle are calculated based on the obstacle parallax information. The step of calculating the position of the obstacle based on the obstacle parallax information includes: The outline of the obstacle is determined in the initial image; A spatial rectangular coordinate system is established based on the initial image, and the x and y values ​​of the center point of the contour are obtained; The first depth value of the contour center point is calculated based on the obstacle parallax information and used as the z value; A second depth value is output by a depth estimation network using one monocular image from the binocular images, and the first depth value is verified based on the second depth value.

2. The method of claim 1, wherein, Before matching the semantic segmentation information with the disparity information, the method further includes: The image segmentation model is trained using disparity information samples and the corresponding label information. The trained image segmentation model segments the disparity information in the initial image into foreground disparity information and background disparity information.

3. The method according to claim 1, characterized in that, The step of matching the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information includes: Determine whether the corresponding pixel in the semantic segmentation map for each pixel in the disparity map belongs to the semantic category of obstacle; If so, the pixel is retained, and obstacle parallax information is determined based on all the retained pixels.

4. The method according to claim 1, characterized in that, The method further includes: A training set is constructed using image samples and the corresponding depth labels to train a predefined convolutional neural network. The initial image is input into the trained depth estimation network, which outputs a second depth value for the center point of the contour.

5. The method according to claim 4, characterized in that, After outputting the second depth value of the center point of the contour, the method further includes: Calculate the difference between the first depth and the second depth values, and determine whether the difference is less than a preset value; If the value is less than the specified depth, the first depth value is determined to be successfully verified, and the position of the obstacle is output based on the x, y, and z values ​​of the center point of the contour.

6. The method according to claim 1, characterized in that, Calculating the size of the obstacle based on the obstacle parallax information includes: Calculate the coordinates of at least one edge point of the obstacle in the spatial rectangular coordinate system; The size of the obstacle is calculated based on the outline of the obstacle and the coordinates of the at least one edge point.

7. An obstacle detection device, characterized in that, include: The acquisition module is used to acquire at least one initial image for the current scene; The extraction module is used to extract disparity information and semantic segmentation information from the initial image. The disparity information includes foreground disparity information and background disparity information. The semantic segmentation information refers to classifying objects of different categories in the initial image. The method of extracting the semantic segmentation information includes: extracting edge information and depth information from the initial image, and performing feature fusion based on the edge information and depth information to obtain the semantic segmentation information in the initial image. A matching module is used to match the semantic segmentation information with the disparity information to determine the obstacle disparity information in the disparity information, including: matching the semantic segmentation information with the foreground disparity information to determine whether there is an obstacle in the foreground; If it exists, then obtain the obstacle disparity information from the foreground disparity information; If it does not exist, the semantic segmentation information is matched with the background disparity information to obtain the obstacle disparity information in the background disparity information; The calculation module is used to calculate the position and size of the obstacle based on the obstacle parallax information; the calculation of the position of the obstacle based on the obstacle parallax information includes: The outline of the obstacle is determined in the initial image; A spatial rectangular coordinate system is established based on the initial image, and the x and y values ​​of the center point of the contour are obtained; The first depth value of the contour center point is calculated based on the obstacle parallax information and used as the z value; A second depth value is output by a depth estimation network using one monocular image from the binocular images, and the first depth value is verified based on the second depth value.

8. A lawnmower robot, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the obstacle detection method as described in any one of claims 1-6.

9. A storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the obstacle detection method as described in any one of claims 1-6.

Images