Equipment obstacle avoidance method, self-moving equipment and storage medium

Abstract

The invention provides an equipment obstacle avoidance method, self-moving equipment and a storage medium. The method comprises the following steps: determining a working area and an obstacle from an acquired first image; based on an operation object of the operation area, a detection area is determined, and the detection area is an area not including the operation object in the operation area; determining the type of an obstacle object corresponding to the detection area based on the image positions of the obstacle and the detection area in the first image; determining the distance between the self-moving device and the obstacle object based on the position information of the obstacle object in the world coordinate system; and according to the distance and the type of the obstacle object, determining an obstacle avoidance route of the self-moving device, and executing a job task based on the obstacle avoidance route. According to the method, the accuracy of identifying the low obstacle can be improved.

Description

Technical Field

[0001] The present application relates to the field of self-moving equipment, and in particular to a device obstacle avoidance method, a self-moving equipment and a storage medium. Background Art

[0002] At present, self-propelled devices mainly use binocular cameras, TOF (Time of Flight) cameras or laser radars to measure the distance to obstacles. Among them, the solution of using binocular cameras or TOF cameras for distance measurement is relatively expensive and not universal. In the process of using laser radar for distance measurement, the scanning angle of the laser radar is limited. For low obstacles, if their height is lower than the scanning range of the laser beam of the laser radar, it is difficult to accurately capture the position of the obstacle. Summary of the invention

[0003] The embodiments of the present application disclose a device obstacle avoidance method, a self-moving device and a storage medium, which solve the technical problems of low recognition accuracy and high recognition cost of low obstacles.

[0004] The present application provides a device obstacle avoidance method, which is applied to a self-moving device, and the method includes: determining a working area and an obstacle from an acquired first image; determining a detection area based on a working object in the working area, and the detection area is an area within the working area that does not include the working object; determining the type of obstacle object corresponding to the detection area based on the image position of the obstacle and the detection area in the first image; determining the distance between the self-moving device and the obstacle object based on the position information of the obstacle object in the world coordinate system; determining an obstacle avoidance route of the self-moving device according to the distance and the type of the obstacle object, and performing the working task based on the obstacle avoidance route.

[0005] In some optional embodiments, the method further includes: using a preset image segmentation model to recognize the first image, to obtain a first mask image corresponding to the working area, a second mask image corresponding to the obstacle, a detection frame corresponding to the obstacle, and the type of the obstacle.

[0006] In some optional embodiments, the detection area is determined based on the working object in the working area, including: determining the inner contour points of the working area according to the first mask image; determining the area formed by the convex hull contour excluding the working object according to the inner contour points; and fitting the minimum circumscribed rectangle of the area formed by the convex hull contour to obtain the detection area.

[0007] In some optional embodiments, determining the type of obstacle object corresponding to the detection area based on the image positions of the obstacle and the detection area in the first image includes: determining a first position of the detection area in the first image, and determining a second position of the second mask image in the first image; if the first position is within a preset range of the second position, determining the type of the obstacle object to be the type of the obstacle; if the first position is not within the preset range of the second position, determining the type of the obstacle object to be an unknown type.

[0008] In some optional embodiments, the self-moving device includes a shooting device, and the method also includes determining the position information of the obstacle object in the world coordinate system, including: if the type of the obstacle object is an unknown type, determining the first image coordinates of the convex hull contour of the obstacle object in the image coordinate system; converting the first image coordinates into first camera coordinates according to the intrinsic parameter matrix of the shooting device; converting the first camera coordinates into first world coordinates in the world coordinate system according to the extrinsic parameter matrix of the shooting device, to obtain the position information of the obstacle object.

[0009] In some optional embodiments, the self-moving device includes a shooting device, and the method also includes determining the position information of the obstacle object in the world coordinate system, including: if the type of the obstacle object is the type of the obstacle, determining the second image coordinates of the detection frame in the image coordinate system; converting the second image coordinates into second camera coordinates according to the intrinsic parameter matrix of the shooting device; converting the second camera coordinates into second world coordinates in the world coordinate system according to the extrinsic parameter matrix of the shooting device, to obtain the position information of the obstacle object.

[0010] In some optional embodiments, the method further includes: determining the orientation information of the working area and the obstacle based on the first mask map and the second mask map; if it is determined based on the orientation information that the obstacle is located above the working area, fitting the first boundary line of the obstacle based on the convex hull contour points of the obstacle; and calculating the distance between the first boundary line and the self-moving device as the distance between the self-moving device and the obstacle.

[0011] In some optional embodiments, the method further includes: determining the outer contour points of the working area based on the first mask map; if the outer contour points coincide with the boundary of the second mask map, determining that there are overlapping target outer contour points; generating a second boundary line based on the target outer contour points; and calculating the distance between the second boundary line and the self-moving device as the distance between the self-moving device and the obstacle.

[0012] The present application also provides a self-moving device, which includes a processor and a memory, and the processor is used to implement the device obstacle avoidance method when executing a computer program stored in the memory.

[0013] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the device obstacle avoidance method is implemented.

[0014] In the device obstacle avoidance method provided in the present application, the working area and the obstacle are determined from the acquired first image, and the detection area that does not include the working object in the working area is determined based on the working object in the working area. By determining the detection area in the working area, the detection accuracy of the working area can be improved. Then, based on the image position of the obstacle and the detection area in the first image, the type of obstacle object corresponding to the detection area is determined, which improves the recognition accuracy of low obstacles and provides a basis for subsequent planning of obstacle avoidance routes based on the type of obstacle objects. Based on the position information of the obstacle object in the world coordinate system, the distance between the self-moving device and the obstacle object is determined, which provides a basis for subsequent planning of obstacle avoidance routes based on the distance and improves the accuracy of route planning. Finally, according to the distance and the type of obstacle object, the obstacle avoidance route of the self-moving device is determined, and the working task is performed based on the obstacle avoidance route. The above method can improve the accuracy of identifying low obstacles and avoid the problem of affecting the progress of work when there are low obstacles in the working area to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the structure of a self-moving device provided in an embodiment of the present application.

[0016] Figure 2 It is a flow chart of the device obstacle avoidance method provided in an embodiment of the present application.

[0017] Figure 3 It is a processing diagram of the image segmentation model in the embodiment of the present application.

[0018] Figure 4 It is a flow chart for determining the detection area provided in an embodiment of the present application.

[0019] Figure 5 It is a schematic diagram of the detection area provided in the embodiment of the present application.

[0020] Figure 6 It is a flow chart for determining location information provided in an embodiment of the present application.

[0021] Figure 7 It is a schematic diagram of the camera coordinate system and the image coordinate system provided in the embodiment of the present application. DETAILED DESCRIPTION

[0022] To facilitate understanding, some illustrations of concepts related to the embodiments of the present application are given by way of example for reference.

[0023] It should be noted that in this application, "at least one" means one or more, and "more than one" means two or more than two. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The terms "first", "second", "third", "fourth", etc. (if any) in the specification, claims and drawings of this application are used to distinguish similar objects, rather than to describe a specific order or sequence.

[0024] The current self-moving devices have the ability to identify obstacles. By identifying obstacles, they can avoid structural damage caused by collisions with obstacles. Self-moving devices mainly use binocular cameras, TOF (Time of Flight) cameras or laser radars to measure the distance to obstacles. At present, there may be some low obstacles in the moving scenes of self-moving devices. The solution of using binocular cameras or TOF cameras for ranging can identify such low obstacles, but the cost of binocular cameras or TOF cameras is high and not universal. In addition, low obstacles are in the scanning blind spot of the laser radar. In the process of using laser radar for ranging, if the height of the obstacle is lower than the scanning range of the laser beam of the laser radar, it is difficult to accurately capture the position of the obstacle.

[0025] Therefore, in order to solve the technical problems of low recognition accuracy and high recognition cost of low obstacles, the embodiments of the present application provide a device obstacle avoidance method, a self-moving device and a storage medium, which combine an image processing algorithm with a monocular ranging algorithm, reduce the cost of identifying low obstacles, and improve the recognition accuracy of low obstacles. The device obstacle avoidance method of the present application is applied to a self-moving device, and the structure of the self-moving device is first described below.

[0026] Figure 1 Schematic diagram of the structure of the self-moving device provided in the embodiment of the present application. Figure 1 As shown, in the embodiment of the present application, the self-moving device 100 may include a body and a shooting device 10, a memory 11, a processor 12, a power supply 13, a control device 14, a sensor 15, an operation device 16, a communication module 17, a positioning module 18, a driving wheel 19 and a bus 20 arranged on the body. The processor 12 is coupled to the shooting device 10, the memory 11, the power supply 13, the control device 14, the sensor 15, the operation device 16, the communication module 17, the positioning module 18, and the driving wheel 19 through the bus 20.

[0027] The shooting device 10 may be an image acquisition device with zoom (manual zoom or automatic zoom) and auto-focus functions, such as a monocular camera, an analog camera, a high-definition camera, or a network camera, etc. The shooting device 10 is used to obtain image data or video data of the surrounding environment of the mobile device 100 during its movement.

[0028] The memory 11 may be used to store an operating system, computer programs, and various types of data, images, etc. For example, the memory 11 stores a program corresponding to the device obstacle avoidance method provided in an embodiment of the present application.

[0029] The processor 12 provides computing and control capabilities to support the operation of the entire self-moving device. For example, the processor 12 is used to execute a computer program stored in the memory 11 to implement the device obstacle avoidance method provided in the embodiment of the present application.

[0030] The power source 13 is used to supply power to the mobile device 100. In an embodiment of the present application, the power source 13 may include a battery pack composed of a plurality of battery cells.

[0031] The control device 14 is used to control the movement and behavior of the mobile device 100. In an embodiment of the present application, the control device 14 can implement functions such as movement control and logic control.

[0032] The sensor 15 is used to obtain data for the mobile device 100, such as obtaining environmental data and data related to the mobile device 100. In an embodiment of the present application, the sensor 15 may include one or more of a collision sensor, a current sensor, a voltage sensor, a rain detection sensor, a laser radar, and an ultrasonic sensor.

[0033] The working device 16 is used to achieve corresponding working functions, such as mowing, road cleaning, spraying pesticides, etc. In one embodiment of the present application, the working device 16 may include a driving mechanism such as a motor and a hydraulic cylinder, and a cutter disc including a blade. In one embodiment of the present application, the motor can drive the blade to move to complete the mowing task. The motor can adjust the height and speed of the blade when mowing by controlling the movement of the blade.

[0034] The communication module 17 is used to realize communication between the mobile device 100 and other devices. In one embodiment of the present application, the communication module 17 can exchange data with other devices based on wired communication and / or wireless communication. The above-mentioned wireless communication can include one or more combinations of communication modes such as Bluetooth communication, Wi-Fi communication, and Near Field Communication (NFC).

[0035] The positioning module 18 is used to determine the position and movement direction of the mobile device 100. In one embodiment of the present application, the positioning module 18 may include a global positioning system (GPS), an inertial measurement unit (IMU), and the like.

[0036] The driving wheel 19 is used to realize the movement of the self-moving device 100. In one embodiment of the present application, the driving wheel 19 can realize the movement of the self-moving device 100 on the target planning trajectory according to the control of the control device 14. In one embodiment of the present application, the self-moving device 100 may include a driving wheel and a passive wheel, and the driving wheel may also include a left driving wheel and a right driving wheel.

[0037] The bus 20 is at least used to provide a channel for mutual communication between the shooting device 10, memory 11, processor 12, power supply 13, control device 14, sensor 15, operating device 16, communication module 17, positioning module 18, and driving wheel 19 in the mobile device 100.

[0038] In this embodiment, the memory 11 may be an internal memory of the mobile device 100, that is, a memory built into the mobile device 100. In other embodiments, the memory 11 may also be an external memory of the mobile device 100, that is, a memory externally connected to the mobile device 100.

[0039] It should be understood that the memory 11 may include a program storage area and a data storage area. The program storage area may be used to store an operating system, an application required by at least one method (such as a device obstacle avoidance method, etc.), etc.; the data storage area may store data created based on the use of the mobile device 100, etc. In addition, the memory 11 may include a volatile memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other storage devices.

[0040] In some embodiments of the present application, the memory 11 is used to store program codes and various data, and to achieve high-speed and automatic access to programs or data during the operation of the mobile device 100 .

[0041] In some embodiments of the present application, the processor 12 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or a processor, or any other conventional processor, etc.

[0042] The autonomous mobile device 100 may be a semi-autonomous mobile device or a fully autonomous mobile device, and may be a smart lawn mower, a sweeping robot, a cleaning robot, etc.

[0043] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the self-mobile device 100 to which the scheme of the present application is applied. The specific self-mobile device 100 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, the self-mobile device 100 may also include a display screen, a radio chipset, an early warning device (for example, a sound alarm, a photoelectric alarm, or an audible and visual alarm, etc.), and other structures.

[0044] Figure 2 is a flow chart of the device obstacle avoidance method provided in the embodiment of the present application. Figure 2 As shown, the device obstacle avoidance method provided in the embodiment of the present application is applied to a self-moving device (such as Figure 1 According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0045] Step S201: determine the working area and obstacles from the acquired first image.

[0046] In some embodiments of the present application, the self-moving device includes a shooting device, which can be a monocular camera. Monocular cameras have the advantages of low cost and low computational complexity. During the movement of the self-moving device, the shooting device collects images of the surrounding environment to obtain environmental information through the images. The image collected by the shooting device is used as the first image.

[0047] In addition, a segmentation network and a post-processing algorithm may be pre-deployed in the mobile device. The segmentation network may also be called an image segmentation network, an image segmentation model, a region segmentation model, etc. It is a neural network that can classify each pixel in the image, determine the category of each point, and thus perform region segmentation.

[0048] The image segmentation model can be a neural network model based on a convolutional neural network (CNN) model, a recurrent neural network (RNN) model, a long short-term memory network (LSTM) model, a deep neural network (DNN) model, a generative pre-training transformer (GPT) large model, etc.

[0049] Optionally, the image segmentation model can be a yolov8 model. The yolov8 model can use an image encoder to encode the image to obtain an image embedding, where the image embedding is a dense vector representation of the image, that is, the feature of the image. The yolov8 model can segment an image into multiple regions based on image features, and each region can correspond to an instance. The yolov8 model can use a mask to represent each segmented region. The above is just an example, not limited to the yolov8 model, and can also be other image segmentation models, which are not limited to the embodiments of the present application.

[0050] The post-processing algorithm may be an algorithm that further processes the output results of the image segmentation model (e.g., each area represented by a mask). The post-processing algorithm may include, but is not limited to: a denoising algorithm, a detail enhancement algorithm, an image binarization algorithm, an edge detection algorithm (e.g., the Canny operator), and a contour finding algorithm (e.g., the cv2.findContours function in OpenCV), etc.

[0051] In some embodiments of the present application, after the self-mobile device acquires the first image through the shooting device, the self-mobile device calls the image segmentation model, uses the image segmentation model to perform image processing on the first image, and obtains the segmentation result output by the image segmentation model. The segmentation result is represented by a mask, and masks with different pixel values ​​are used to represent different areas in the first image. Based on the pixel values ​​of the first image, the working area and obstacles in the first image can be determined. The working area can be the area where the self-mobile device needs to perform the working task. For example, when the self-mobile device is a lawn mower, the working area can be a lawn. Obstacles can be people, trees, small animals, etc.

[0052] Combination Figure 3 Describes the segmentation result generated by the image segmentation model after processing the first image. Figure 3 As shown, the first image is input into the image segmentation model to obtain the segmentation result. The segmentation result is represented by a mask, Q1 represents the tree mask obtained by segmentation, Q2 represents the human mask obtained by segmentation, Q3 represents the animal mask obtained by segmentation, Q4 represents the miscellaneous tree pile mask obtained by segmentation, and Q5 represents the lawn mask. Among them, the working area includes the lawn mask, and the lawn mask is recorded as the first mask map. The obstacles include tree masks, human masks, animal masks, and miscellaneous tree pile masks, and the masks corresponding to these obstacles are recorded as the second mask map.

[0053] In addition, the image segmentation model also outputs the detection box corresponding to the obstacle and the type of obstacle.

[0054] In one example, after the image segmentation model determines the tree mask, the tree mask is identified, and the type of obstacle corresponding to the tree mask is obtained as a tree. A detection frame is marked for the tree to determine the location and range of the tree, which is convenient for subsequent processing and analysis. After the image segmentation model determines the person mask, the person mask is identified, and the type of obstacle corresponding to the person mask is obtained as a person. A detection frame is marked for the person to determine the location and range of the person, which is convenient for subsequent processing and analysis. In addition, the detection frame can also be used to track and identify dynamically moving people in real time.

[0055] Step S202: determining a detection area based on the working object in the working area.

[0056] In some embodiments of the present application, the working area is the area where the automatic device performs the work. For example, if the automatic device is a lawn mower and the work task is mowing the lawn, the working area is the area where the lawn is located (e.g. Figure 3The area indicated by Q5 shown in the figure) and the working object corresponding to the working area are lawns. In order to detect whether there are obstacles that affect the operation of the self-moving equipment in the working area, the inner contour points in the mask corresponding to the working area are extracted using a post-processing algorithm to obtain a detection area. The detection area is an area that does not include the working object. In addition, the specific determination process of the detection area can refer to the following Figure 4 The embodiment shown.

[0057] In one example, if Figure 3 As shown, the animal corresponding to Q3 runs to the area where Q5 is located. If there is an area corresponding to Q3 in the area corresponding to Q5, then the area corresponding to Q3 in the area corresponding to Q5 is the detection area.

[0058] In another example, the operation area is the area where the lawn is located, and the operation object is the lawn. If there is a small open space or a pond in the operation area, then the open space or the pond is the detection area that does not contain grass.

[0059] Not limited to the above examples, the detection area may be any area within the operating area that does not contain the operating object, and the present application does not limit the area type corresponding to the detection area.

[0060] Step S203: determining the type of obstacle object corresponding to the detection area based on the image positions of the obstacle and the detection area in the first image.

[0061] In some embodiments of the present application, an image coordinate system is established based on the first image. The image coordinate system can be used to determine the image coordinates of the detection area in the image coordinate system, thereby determining the first position of the detection area in the first image according to the image coordinates corresponding to the detection area. The image coordinate system can be used to determine the image coordinates of the second mask image in the image coordinate system, thereby determining the second position of the obstacle in the first image according to the image coordinates corresponding to the second mask image.

[0062] In some embodiments of the present application, after determining the first position of the detection area and the second position of the obstacle, the first position is matched with the second position. If the first position is within the preset range of the second position, or if the second position is within the preset range of the first position, indicating that the obstacle and the detection area are located at the same position, the type of obstacle object corresponding to the detection area is determined to be the type of obstacle. For example, if the obstacle is an animal, the obstacle object corresponding to the detection area is an animal.

[0063] If the first position is not within the preset range of the second position, or the second position is not within the preset range of the first position, indicating that the obstacle is located at a different position from the detection area, then the type of obstacle object corresponding to the detection area is determined to be an unknown type. Since obstacles are the product of the output of the image segmentation model, and the obstacles that can be segmented by the image segmentation model depend on the training set. Therefore, obstacle objects of unknown types may be obstacles that are not covered in the training set, such as temporary tents, parasols, and other objects that temporarily exist in the working area. Obstacle objects of unknown types may also be objects that do not need to be marked in the training set, such as small stones, open spaces, and other objects that do not affect the operation of self-moving equipment.

[0064] Step S204: determining the distance between the mobile device and the obstacle object based on the position information of the obstacle object in the world coordinate system.

[0065] In some embodiments of the present application, a world coordinate system is established with the wheels of the self-moving device as the origin. The image coordinates of the obstacle object in the image coordinate system are obtained, the image coordinates corresponding to the obstacle object are converted into camera coordinates according to the camera parameters of the shooting device, the camera coordinates are converted into world coordinates in the world coordinate system according to the camera parameters, and the position information of the obstacle object in the world coordinate system is obtained according to the world coordinates. The distance of the obstacle object from the self-moving device can be obtained according to the position information, which is convenient for subsequent planning of obstacle avoidance routes and controlling the self-moving device to avoid obstacles.

[0066] In addition, for different types of obstacle objects, different methods can be used to calculate the location information of the obstacle object to improve the accuracy of distance calculation. The specific calculation method can be found in the following Figure 6 The embodiment shown.

[0067] In other embodiments of the present application, there may be obstacles above the working area. Image features of the first mask image and the second mask image are extracted, and the image features include edge features, texture features, and shape features. According to the image features and the image coordinates of the first mask image and the second mask image in the image coordinate system, the relative position relationship between the first mask image and the second mask image is calculated, and the relative position relationship includes the distance relationship and the angle relationship between the first mask image and the second mask image. The orientation information between the working area and the obstacle is obtained according to the relative position relationship.

[0068] If it is determined based on the orientation information that the obstacle is located above the working area, the first boundary line of the obstacle is fitted according to the convex hull contour points of the obstacle. The convex hull can be understood as: for a set D, the set of all convex combinations of any finite number of points in D is called the convex hull of D. The image coordinates of the first boundary line in the image coordinate system are obtained. Based on the camera parameters, the image coordinates corresponding to the first boundary line are converted to world coordinates in the world coordinate system. The distance between the first boundary line and the self-moving device is determined according to the world coordinates corresponding to the first boundary line, thereby obtaining the distance between the obstacle above the working area and the self-moving device.

[0069] In yet another embodiment of the present application, there may be obstacles on the boundary of the working area. Determine the outer contour points of the first mask image, that is, determine the outer contour points corresponding to the working area. Obtain the boundary of the second mask image. If the outer contour points coincide with the boundary of the second mask image, determine that there are overlapping target outer contour points. Generate a second boundary line based on the target outer contour points. Obtain the image coordinates of the second boundary line in the image coordinate system. Based on the camera parameters, convert the image coordinates corresponding to the second boundary line into world coordinates in the world coordinate system. Determine the distance between the second boundary line and the self-moving device based on the world coordinates corresponding to the second boundary line, thereby obtaining the distance between the obstacle on the boundary of the working area and the self-moving device.

[0070] Step S205, determining an obstacle avoidance route for the mobile device according to the distance and the type of the obstacle object, and executing the operation task based on the obstacle avoidance route.

[0071] In some embodiments of the present application, the obstacle object is located in the working area, and the obstacle object may be an obstacle of unknown type or an obstacle of known type.

[0072] In some embodiments of the present application, for an unknown type of obstacle object, a corresponding obstacle avoidance route can be determined according to the corresponding distance. According to the image coordinates corresponding to the detection area where the unknown type of obstacle object is located, the size of the detection area where the unknown type of obstacle object is located is determined and recorded as the area value. The distance between the unknown type of obstacle object and the self-moving device is recorded as the first distance.

[0073] In one example, if the area value of the region is less than a preset area threshold, the self-moving device can be controlled to continue driving. For example, if the unknown type of obstacle is a small piece of open space and does not affect the operation of the self-moving device, the self-moving device can ignore the unknown type of obstacle object and continue to perform the operation task.

[0074] If the area value of the region is greater than the preset area threshold, and the first distance is greater than the preset distance threshold, a first obstacle avoidance route of the self-moving device is determined, and the first obstacle avoidance route may include a route that turns at a preset angle. The self-moving device is controlled to start turning based on the first obstacle avoidance route to avoid the obstacle object and continue to perform the task.

[0075] If the area value of the region is greater than the preset area threshold, and the first distance is less than or equal to the preset distance threshold, a second obstacle avoidance route of the self-moving device is determined, and the second obstacle avoidance route may include a route that retreats a certain preset distance, and / or a route that turns at a preset angle. The self-moving device is controlled to retreat a certain preset distance based on the second obstacle avoidance route, and then turn to avoid the obstacle object and continue to perform the task. Alternatively, the self-moving device can be controlled to stop moving and an alarm can be issued to prompt the user that there is a large obstacle object that is difficult to avoid.

[0076] In some embodiments of the present application, for an obstacle object of a known type, a first type of obstacle avoidance route can be determined according to a corresponding distance. The distance between the obstacle object of the known type and the self-moving device is recorded as the second distance.

[0077] In one example, if the known type of obstacle object is a dynamically moving obstacle object (such as a person or an animal), and the second distance is greater than a preset distance threshold, the current route will not be changed. If the known type of obstacle object is a dynamically moving obstacle, and the second distance is less than or equal to the preset distance threshold, the vehicle can travel according to the first obstacle avoidance route or the second obstacle avoidance route to avoid the obstacle object and complete the task.

[0078] If the known type of obstacle object is a static obstacle object (such as a stone or a tree), and the second distance is greater than the preset distance threshold, the current route will not be changed. If the known type of obstacle object is a static obstacle, and the second distance is less than or equal to the preset distance threshold, the vehicle can travel according to the first obstacle avoidance route to avoid the obstacle object and complete the task.

[0079] In other embodiments of the present application, there may be obstacles above or at the border of the work area. For obstacles that are static above the work area, the current route may not be changed. For obstacles that are dynamically moving above the work area, the first obstacle avoidance route or the second obstacle avoidance route may be used to avoid the obstacle and complete the work task. For obstacles at the border of the work area, the first obstacle avoidance route may be used to turn at a preset angle to avoid the obstacle and complete the work task.

[0080] In the above embodiment, the working area and the obstacle are determined from the acquired first image, and the detection area that does not include the working object in the working area is determined based on the working object in the working area. By determining the detection area in the working area, the detection accuracy of the working area can be improved. Then, based on the image position of the obstacle and the detection area in the first image, the type of obstacle object corresponding to the detection area is determined, which improves the recognition accuracy of low obstacles and provides a basis for subsequent planning of obstacle avoidance routes based on the type of obstacle objects. Based on the position information of the obstacle object in the world coordinate system, the distance between the self-moving device and the obstacle object is determined, which provides a basis for subsequent planning of obstacle avoidance routes based on the distance and improves the accuracy of route planning. Finally, according to the distance and the type of obstacle object, the obstacle avoidance route of the self-moving device is determined, and the working task is performed based on the obstacle avoidance route. The above method can improve the accuracy of identifying low obstacles and avoid the problem of affecting the progress of work when there are low obstacles in the working area to a certain extent.

[0081] Figure 4 is a flow chart for determining the detection area provided in the embodiment of the present application. Figure 4 As shown, the post-processing algorithm is used to determine the detection area based on the working objects in the working area. The specific determination process of the detection area includes the following steps:

[0082] Step S401, determining inner contour points of the operation area according to the first mask image.

[0083] In some embodiments of the present application, the first mask image is a mask image corresponding to the operating area. The inner contour points in the first mask image can be extracted using a post-processing algorithm. The post-processing algorithm may include, but is not limited to: a denoising algorithm, a detail enhancement algorithm, an image binarization algorithm, an edge detection algorithm (e.g., a Canny operator), and a contour search algorithm (e.g., the cv2.findContours function in OpenCV).

[0084] Step S402: determining, based on the inner contour points, an area formed by the convex hull contour that does not include the work object.

[0085] In some embodiments of the present application, the convex hull contour is the outer contour corresponding to the area where the set of points where the convex hull is located is located. The convex hull contour is determined according to the inner contour points of the operation area, that is, the area surrounded by the convex hull contour is determined, and the area does not include the operation object.

[0086] Step S403, fitting the minimum circumscribed rectangle of the area formed by the convex hull contour to obtain the detection area.

[0087] In some embodiments of the present application, after the area formed by the convex hull contour is determined, the shape of each convex hull contour is different. In order to facilitate subsequent analysis, the detection area can be obtained by fitting the minimum circumscribed rectangle of the area formed by the convex hull contour. The structure of the detection area is a rectangular frame structure.

[0088] Combination Figure 5 The process of determining the detection area is described by: The first image is input into the image segmentation model to obtain the first mask image, and the inner contour points of the first mask image are extracted using the post-processing algorithm. Figure 5 The boundary points of the area where the table is located in the lawn shown are the inner contour points of the first mask image. The area formed by the convex hull contour that does not include the work object is determined based on the inner contour points, such as Figure 5 As shown in , the work object is grass, then the area not including grass is the area where the table is located, the area where the display board is located, the area where the cat is located, and the area where the object placed on the lawn is located. The minimum circumscribed rectangle of the area formed by the fitting convex hull contour. Figure 5 As shown in ①, the area formed by fitting the convex hull contour corresponding to the table is obtained to obtain the corresponding rectangular frame of the table. Figure 5 As shown in ②, the area formed by fitting the convex hull contour corresponding to the display board is obtained to obtain the corresponding rectangular frame of the display board. Figure 5 As shown in ③, the area formed by fitting the convex hull contour corresponding to the cat is obtained to obtain the rectangular frame corresponding to the cat. Figure 5 As shown in ④, the area formed by fitting the convex hull contours corresponding to other objects on the lawn is used to obtain the corresponding rectangular frames of other objects.

[0089] In the above embodiment, the first mask image is analyzed by a post-processing algorithm to obtain the detection area, which provides a basis for subsequently determining obstacles and specification obstacle avoidance routes in the operating area based on the detection area, and to a certain extent increases the accuracy and efficiency of determining low obstacles.

[0090] Figure 6 is a flow chart for determining location information provided by an embodiment of the present application. Figure 6 As shown, the categories of obstacle objects are divided into unknown types and obstacle types. Determining the location information of different obstacle objects includes the following steps:

[0091] Step S601: determining the type of obstacle object corresponding to the detection area based on the image positions of the obstacle and the detection area in the first image.

[0092] Step S602: determine whether the type of the obstacle object is the type of obstacle.

[0093] In some embodiments of the present application, the specific description of steps S601 to S602 can refer to step S203, which will not be repeated here. If the type of the obstacle object is an unknown type, step S603 is executed, and if the type of the obstacle object is an obstacle type, step S606 is executed.

[0094] Step S603: determine the first image coordinates of the convex hull contour of the obstacle object in the image coordinate system.

[0095] In some embodiments of the present application, if the type of the obstacle object is unknown, since the shapes of the unknown obstacle objects are different, in order to improve the accuracy of subsequent analysis of the unknown obstacle objects, the convex hull contour of the obstacle object is used to perform subsequent analysis. The first image coordinates of the convex hull contour of the obstacle object in the image coordinate system are obtained to represent the position of the unknown obstacle object in the working area.

[0096] Step S604: converting the first image coordinates into first camera coordinates according to the intrinsic parameter matrix of the photographing device.

[0097] In some embodiments of the present application, the intrinsic parameter matrix is ​​constructed by the internal parameters of the camera of the shooting device, including focal length, image offset, etc. The conversion relationship between the first image coordinates and the first camera coordinates is determined by the intrinsic parameter matrix.

[0098] Combination Figure 7 Describes the process of transforming the first image coordinates into the first camera coordinates. Figure 7 As shown, X c Y c Z c is the camera coordinate system, the origin of the camera coordinate system is the optical center O, which is the center point of the camera convex lens. X1-O1-Y1 is the image coordinate system on the camera plane of the shooting device, and the positive direction of the image coordinate system is when the X1 axis is facing right and the Y1 axis is facing downward. X2-O2-Y2 is the corresponding image coordinate system on the working area. The line connecting the optical center O and the origin O2 is perpendicular to the working area.

[0099] The conversion between the first image coordinates and the first camera coordinates is expressed by the following formula:

[0100]

[0101] Among them, u and v are the pixel coordinates of the X1 axis and Y1 axis in the image coordinate system, and Z c is the Z coordinate system of the camera c Axis coordinates, that is, Figure 7 Depth OD shown. f x and f y are the pixel focal lengths of the X1 and Y1 axes of the image coordinate system (in pixels).x and c y It is the pixel coordinate of the center point O1 of the image (the point where the optical center of the camera coordinate system forms an image on the image). The depth OD is calculated as follows:

[0102] like Figure 7 The plastic bottle shown is in contact with point Q on the ground. During the distance measurement process, point Q on the plastic bottle corresponds to point Q' on the image (e.g., the first image). The distance from point Q' to point O1 along the y-axis is O1P', where P' is the projection of Q' on the Y1 axis of the image coordinate system. The distance O1P' is divided by the pixel focal length f y , and then find arctan to get angle b′, which means we can get angle b.

[0103] ∠b′=arctan(O1P′ / f y );

[0104] ∠b′=∠b;

[0105] ∠c=∠a+∠b;

[0106]

[0107] OD = OP cos(b);

[0108] Wherein, ∠a is the pitch angle of the camera relative to the ground, and H is the distance between the optical center O and the working area.

[0109] Step S605 , converting the first camera coordinates into first world coordinates in a world coordinate system according to the extrinsic parameter matrix of the shooting device, and obtaining position information of the obstacle object.

[0110] In some embodiments of the present application, the external parameter matrix is ​​used to describe the conversion relationship between the first camera coordinates and the world coordinate system. The wheel center of the self-moving device is set as the origin of the world coordinate system, and the moving direction of the self-moving device is set as X w The positive direction of the axis is set to Y from the right side of the moving direction of the mobile device w The positive direction of the axis, the vertical upward direction from the mobile device is set to Z w The origin of the camera coordinate system is the optical center of the convex lens, and the direction perpendicular to the convex lens is the Z axis. c Positive direction, the right side of the camera is X c Positive direction, the top of the camera is Y c Positive direction.

[0111] In general, the transformation between the first camera coordinates and the world coordinate system mainly includes the rotation of the first camera coordinates around the three axes of the world coordinate system and the translation along the three axes, which can be expressed by the following formula:

[0112] Around X w The rotation matrix of the axis rotation angle θ:

[0113] Around Y w The rotation matrix of the axis rotation angle θ:

[0114] Around Z w The rotation matrix of the axis rotation angle θ:

[0115] In addition, you need to determine the rotation order around the axes of the world coordinate system, for example, around X w , Y w , Z w The total rotation matrix is ​​R = R Z R Y R X , where the axis that rotates first is on the right side of formula R.

[0116]

[0117] Here, s represents the torsion factor.

[0118] In some embodiments of the present application, the external parameter matrix includes a rotation matrix R and a translation matrix T. The conversion between the first camera coordinate system and the world coordinate system can be expressed by the following formula:

[0119]

[0120] Through the above formula, the location information of the obstacle object can be obtained, for example, Figure 7 The location of the plastic bottle on the lawn is shown.

[0121] Step S606: determine the second image coordinates of the detection frame in the image coordinate system.

[0122] In some embodiments of the present application, if the type of the obstacle object is the type of obstacle, the detection frame generated by the image segmentation model can be used as the basis for subsequent analysis to improve the accuracy of subsequent analysis. The second image coordinates of the detection frame in the image coordinate system are obtained to represent the position of the obstacle object of the type of obstacle in the working area.

[0123] Step S607: convert the second image coordinates into second camera coordinates according to the intrinsic parameter matrix of the shooting device.

[0124] In some embodiments of the present application, the intrinsic parameter matrix is ​​constructed by the internal parameters of the camera of the shooting device, including focal length, image offset, etc. The conversion relationship between the second image coordinates and the second camera coordinates is determined by the intrinsic parameter matrix. The conversion between the image coordinates and the camera coordinates can refer to step S604, which will not be repeated here.

[0125] Step S608: convert the second camera coordinates into second world coordinates in the world coordinate system according to the extrinsic parameter matrix of the shooting device to obtain the position information of the obstacle object.

[0126] In some embodiments of the present application, the conversion between the camera coordinates and the world coordinates can refer to step S605, which is not described again here. The second camera coordinates are converted into second world coordinates in the world coordinate system according to the external parameter matrix of the shooting device, and the position of the obstacle object in the working area can be obtained according to the second world coordinates.

[0127] Through the above embodiments, different methods are used to determine the position of the obstacle object in the working area, providing accurate position information for the subsequent planning of the obstacle avoidance route, and improving the accuracy of identifying low obstacles to a certain extent.

[0128] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the device obstacle avoidance method of the present application.

[0129] The computer-readable storage medium may be an internal storage unit of the self-mobile device described in the foregoing embodiments, such as a hard disk or memory of the self-mobile device. The computer-readable storage medium may also be an external storage device of the self-mobile device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the self-mobile device.

[0130] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created based on the use of the mobile device, etc.

[0131] The self-moving device and computer-readable storage medium provided in the above-mentioned embodiments can obtain a first image of the surrounding area through a shooting device when the self-moving device is in operation, so that the self-moving device analyzes the first image to obtain the position of the low obstacle, and then calculates the distance between the self-moving device and the low obstacle according to the measurement algorithm. The obstacle avoidance route is determined according to the distance and the type of obstacle object in the operation area, and the operation task is performed according to the obstacle avoidance route, thereby improving the operation efficiency of the self-moving device.

[0132] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0133] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0134] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A device obstacle avoidance method, applied to a self-moving device, characterized in that: The method comprises: determining a working area and obstacles from the acquired first image; Based on the working object in the working area, determining a detection area, the detection area being an area in the working area that does not include the working object; Determining a type of obstacle object corresponding to the detection area based on image positions of the obstacle and the detection area in the first image; Determine the distance between the self-moving device and the obstacle object based on the position information of the obstacle object in the world coordinate system; An obstacle avoidance route of the self-moving device is determined according to the distance and the type of the obstacle object, and a work task is performed based on the obstacle avoidance route.

2. The device obstacle avoidance method according to claim 1, characterized in that: The method further comprises: The first image is recognized by using a preset image segmentation model to obtain a first mask image corresponding to the working area, a second mask image corresponding to the obstacle, a detection frame corresponding to the obstacle, and the type of the obstacle.

3. The device obstacle avoidance method according to claim 2, characterized in that: The step of determining the detection area based on the operation object in the operation area includes: Determining inner contour points of the operation area according to the first mask map; Determine, according to the inner contour points, an area formed by the convex hull contour excluding the operation object; The minimum circumscribed rectangle of the area formed by the convex hull contour is fitted to obtain the detection area.

4. The device obstacle avoidance method according to claim 2, characterized in that: The determining, based on the image positions of the obstacle and the detection area in the first image, the type of the obstacle object corresponding to the detection area includes: Determining a first position of the detection area in the first image, and determining a second position of the second mask image in the first image; If the first position is within a preset range of the second position, determining that the type of the obstacle object is the type of the obstacle; If the first position is not within the preset range of the second position, it is determined that the type of the obstacle object is an unknown type.

5. The device obstacle avoidance method according to claim 1, characterized in that: The self-moving device includes a shooting device, and the method further includes determining position information of the obstacle object in a world coordinate system, including: If the type of the obstacle object is an unknown type, determining a first image coordinate of a convex hull contour of the obstacle object in an image coordinate system; Converting the first image coordinates into first camera coordinates according to the intrinsic parameter matrix of the shooting device; The first camera coordinates are converted into first world coordinates in a world coordinate system according to the extrinsic parameter matrix of the shooting device to obtain the position information of the obstacle object.

6. The device obstacle avoidance method according to claim 2, characterized in that: The self-moving device includes a shooting device, and the method further includes determining position information of the obstacle object in a world coordinate system, including: If the type of the obstacle object is the type of the obstacle, determining a second image coordinate of the detection frame in the image coordinate system; Converting the second image coordinates into second camera coordinates according to the intrinsic parameter matrix of the shooting device; The second camera coordinates are converted into second world coordinates in a world coordinate system according to the extrinsic parameter matrix of the shooting device to obtain the position information of the obstacle object.

7. The device obstacle avoidance method according to claim 2, characterized in that: The method further comprises: Determining the orientation information of the working area and the obstacle according to the first mask map and the second mask map; If it is determined based on the orientation information that the obstacle is located above the working area, fitting a first boundary line of the obstacle according to the convex hull contour points of the obstacle; The distance between the first boundary line and the self-moving device is calculated as the distance between the self-moving device and the obstacle.

8. The device obstacle avoidance method according to claim 2, characterized in that: The method further comprises: Determining outer contour points of the operation area according to the first mask image; If the outer contour point coincides with the boundary of the second mask image, determining that there is an overlapping target outer contour point; Generate a second boundary line according to the target outer contour point; The distance between the second boundary line and the self-moving device is calculated as the distance between the self-moving device and the obstacle.

9. A self-propelled device, characterized in that: The self-mobile device includes a processor and a memory, and the processor is used to execute a computer program stored in the memory to implement the device obstacle avoidance method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, the device obstacle avoidance method as described in any one of claims 1 to 8 is implemented.

Images