Line laser-based self-propelled device obstacle avoidance method and device, equipment and medium

Abstract

The present disclosure relates to a line laser-based self-walking device obstacle avoidance method and device, equipment and medium, the method comprising: acquiring a line laser recognition image of the self-walking device; determining whether there is a wire obstacle according to the line laser recognition image to obtain a line laser recognition result; and performing a corresponding obstacle avoidance action according to the line laser recognition result. Through this technical solution, accurate identification of wire obstacles is achieved.

Description

Technical Field

[0001] This disclosure relates to the field of cleaning robot control technology, and in particular to a method, apparatus, device and medium for obstacle avoidance of self-propelled devices based on line lasers. Background Technology

[0002] For common obstacles in the home environment, such as shoes, socks, weighing scales, and fans, which have fixed shapes and sizes, the general recognition combined with line laser ranging method can meet most obstacle avoidance needs. That is, image recognition is used to obtain the type and location information of the obstacle, and line laser ranging is used to obtain the location information of the obstacle. The combination of the two can obtain the type of obstacle and more accurate location information.

[0003] For unfolded loose wires, firstly, the wire's posture is not fixed, and it will occupy a large space when partially or fully unfolded, making it difficult to observe completely from the robot's perspective. For wires that cannot be fully observed, especially those with unstable postures, there is a higher probability of creating blind spots. When the machine moves near the blind spot, the risk of interference with the machine's operation increases. Secondly, for thinner wires, due to their lower height, the accuracy of existing line laser ranging solutions cannot meet the requirements for ranging based on changes in obstacle height. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides an obstacle avoidance method and apparatus for self-propelled devices, a self-propelled device and a computer-readable storage medium, for achieving accurate identification of wire-type obstacles.

[0005] According to a first aspect of the present disclosure, a method for obstacle avoidance of a self-propelled device based on line laser is provided, comprising:

[0006] Acquire the line laser recognition image of the self-propelled device;

[0007] The presence of wire-like obstacles is determined based on the line laser recognition image to obtain the line laser recognition result;

[0008] Based on the line laser recognition results, the corresponding obstacle avoidance action is executed.

[0009] In one embodiment, preferably, determining whether a wire-like obstacle exists based on the line laser recognition image to obtain a line laser recognition result includes:

[0010] The line laser recognition image is analyzed to determine whether there are laser breakpoints in each line laser in the image;

[0011] When a laser breakpoint is present, it indicates the presence of a wire-type obstacle.

[0012] When there is no laser breakpoint, it is determined that there are no wire-type obstacles.

[0013] In one embodiment, preferably, the method further includes:

[0014] Obtain image recognition results;

[0015] In response to the presence of a wire-type obstacle indicated by both the image recognition result and the line laser recognition result, the corresponding obstacle avoidance action is executed based on the image recognition result and the line laser recognition result.

[0016] In one embodiment, preferably, based on the image recognition result and the line laser recognition result, a corresponding obstacle avoidance action is performed, including:

[0017] The first accuracy rate of historical image recognition results and the second accuracy rate of historical line laser recognition results are obtained respectively;

[0018] A first weight corresponding to the image recognition result is determined based on the first accuracy rate, and a second weight corresponding to the line laser recognition result is determined based on the second accuracy rate.

[0019] The comprehensive recognition result is determined based on the image recognition result, the first weight, the line laser recognition, and the second weight;

[0020] Based on the comprehensive recognition results, corresponding obstacle avoidance actions are performed on the wire-type obstacles.

[0021] In one embodiment, preferably, the method further includes:

[0022] In response to the image recognition result indicating the presence of a wire-type obstacle and the line laser recognition result indicating the absence of a wire-type obstacle, or in response to the image recognition result indicating the absence of a wire-type obstacle and the line laser recognition result indicating the presence of a wire-type obstacle, the image recognition result and the line laser recognition result are acquired multiple times.

[0023] Based on the image recognition results and line laser recognition results obtained from multiple acquisitions, it was re-determined whether there were any wire-type obstacles.

[0024] In one embodiment, preferably, the image recognition result includes a bounding box corresponding to the obstacle, and the method further includes:

[0025] Based on the coordinates of each laser breakpoint in the line laser recognition image, the first position coordinate information of the wire-type obstacle is determined;

[0026] The image recognition results are semantically segmented to determine whether each pixel in the location box corresponding to the wire-type obstacle belongs to the wire-type obstacle.

[0027] When a pixel belongs to the wire-type obstacle, the pixel is marked to determine the second position coordinate information of the wire-type obstacle;

[0028] Based on the first position coordinate information and the second position coordinate information, the final position coordinate information is determined;

[0029] Based on the final position coordinates, execute the corresponding obstacle avoidance action.

[0030] In one embodiment, preferably, the method further includes:

[0031] Determine the start and end points of the wire-type obstacles;

[0032] A first obstacle avoidance action is performed in the first area where the start and end points of the wire-type obstacle are located, and a second obstacle avoidance action is performed in the second area other than the first area, wherein the avoidance distance of the first obstacle avoidance action is greater than the avoidance distance of the second obstacle avoidance action.

[0033] In one embodiment, preferably, determining the start and end points of the wire-like obstacle includes:

[0034] The image recognition result is identified using a target image recognition algorithm to determine the start and end points of the wire-type obstacle.

[0035] In one embodiment, preferably, determining the start and end points of the wire-like obstacle includes:

[0036] Based on the first position coordinate information of the wire-type obstacle, determine the endpoint position of the wire-type obstacle;

[0037] The endpoint positions are determined as the start and end points of the wire-type obstacles.

[0038] In one embodiment, preferably, determining the start and end points of the wire-like obstacle includes:

[0039] Obtain the multi-line laser recognition results of the self-propelled device, and determine all pixels within the range of the wire-type obstacle based on the multi-line laser recognition results;

[0040] Clustering is performed on all the pixels to determine the pixel breakpoints within the range of the wire-type obstacle.

[0041] The pixel breakpoints are determined as the start and end points of the wire-like obstacles.

[0042] According to a second aspect of the present disclosure, a self-propelled obstacle avoidance device based on line laser is provided, the device comprising:

[0043] The acquisition unit is used to acquire the line laser recognition image of the self-propelled device;

[0044] The result determination unit is used to determine whether there is a wire-like obstacle based on the line laser recognition image, so as to obtain the line laser recognition result;

[0045] The processing unit is used to execute the corresponding obstacle avoidance action based on the line laser recognition result.

[0046] According to a third aspect of the present disclosure, a self-propelled device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor.

[0047] The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of the first aspect embodiments above.

[0048] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in any of the embodiments of the first aspect.

[0049] The technical solutions provided by the embodiments of this disclosure may include the following beneficial effects:

[0050] In this invention, the existence of wire-type obstacles can be identified through image recognition and line laser recognition. Specifically, based on image recognition, the coordinates of wire-type obstacles are further identified through pixel annotation to obtain a more accurate range of wires. At the same time, the coordinates of wire-type obstacles are determined by the laser breakpoint of the line laser. The image recognition results and line laser recognition results are used to perform avoidance operations, thereby improving the accuracy of wire range judgment and avoidance actions.

[0051] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0052] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0053] Figure 1 This is a schematic diagram of the structure of a self-propelled device according to an exemplary embodiment.

[0054] Figure 2 This is a schematic diagram of the structure of a centerline laser module in a self-propelled device according to an exemplary embodiment.

[0055] Figure 3This is a flowchart illustrating an obstacle avoidance method for a self-propelled device based on a line laser, according to an exemplary embodiment.

[0056] Figure 4 This is a schematic diagram of a line laser image acquired by a self-propelled device according to an exemplary embodiment.

[0057] Figure 5 This is a schematic diagram of a line laser image acquired by another self-propelled device according to an exemplary embodiment.

[0058] Figure 6 This is a schematic diagram of a line laser image acquired by another self-propelled device according to an exemplary embodiment.

[0059] Figure 7 This is a schematic diagram of an image recognition structure according to an exemplary embodiment.

[0060] Figure 8 This is a schematic diagram of another image recognition structure according to an exemplary embodiment.

[0061] Figure 9 This is a block diagram illustrating a line laser-based obstacle avoidance device for a self-propelled walking device, according to an exemplary embodiment. Detailed Implementation

[0062] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0063] This disclosure provides a possible application scenario for a self-propelled device 100, such as a robotic vacuum cleaner, robotic mop, vacuum cleaner, lawnmower, etc. In some embodiments, such as in this embodiment... Figure 1As shown, taking a household robotic vacuum cleaner as an example, during its operation, the robot can clean according to a preset route or an automatically planned route. However, it will inevitably get stuck in some places, such as a chair or a table. In this case, the robot can identify the obstacle through a cloud server, a local server, or its own storage system, and mark the location as an obstacle. The next time it approaches that location, it will automatically avoid the obstacle. In this embodiment, the robot can be equipped with a touch-sensitive display or controlled via a mobile terminal to receive user input commands. The self-propelled device can be equipped with various sensors, such as buffers, cliff sensors, ultrasonic sensors, infrared sensors, magnetometers, accelerometers, gyroscopes, odometers, etc. (the specific structure of each sensor is not described in detail; any of the above-mentioned sensors can be used in this self-propelled device). The robot can also be equipped with wireless communication modules such as WIFI and Bluetooth modules to connect with a smart terminal or server and receive operation commands transmitted by the smart terminal or server through the wireless communication module.

[0064] An optional embodiment of this disclosure provides a line laser module, and another optional embodiment provides a self-moving device, wherein the line laser module is applied to the self-moving device, specifically, the self-moving device is an intelligent cleaning device, such as a robot vacuum cleaner, a robot mop, a floor polisher, or a lawn mower. For ease of description, this embodiment uses a robot vacuum cleaner as an example to describe the technical solution of this disclosure.

[0065] like Figure 2 As shown, in an optional embodiment of this disclosure, the self-moving device may include: a device body, a sensing system, a control system, a drive system, a cleaning system, an energy system, and a human-machine interaction system. These systems coordinate with each other to enable the self-moving device to move autonomously to achieve its cleaning function. The functional components constituting these systems are integrated within the device body.

[0066] The main body of the device has an approximately circular shape (both front and back are circular), but it may also have other shapes, including but not limited to an approximately D-shaped shape with a circular front and back. The sensing system includes a line laser module located above or to the side of the main body of the device. The main control unit of the control system is connected to the line laser module and performs functional control of the self-moving device based on the sensing results of the line laser module.

[0067] In the publicly provided embodiments, the specific location of the line laser module on the device body is not limited. For example, it can be, but is not limited to, the front, rear, left, right, top, middle, and bottom of the device body. Furthermore, the line laser module is positioned at the middle, top, or bottom of the device body in the height direction.

[0068] In some of the embodiments provided in this disclosure, the self-moving device moves forward to perform a task. In order to better detect the environmental information ahead, a line laser module is disposed on the front side of the device body; the front side is the side that the device body faces during the forward movement of the self-moving device.

[0069] like Figure 2 As shown, the line laser module includes a main body 140 and a first image acquisition component. The first image acquisition component includes a first camera 120 disposed on the main body 140, at least one pair of line laser emitting devices 110, and a first image processing module. The pair of line laser emitting devices 110 are located on both sides of the first camera 120 and are used to emit line lasers with linear projection. The first camera 120 and the line laser emitting devices 110 cooperate with each other to acquire a first environmental image. The first image processing module can obtain obstacle distance information based on the first environmental image. The line laser emitting devices 110 are movably connected to the main body 140, and / or the main body 140 is a movable structure, so that the azimuth and rotation angles of the line laser emitting devices 110 are adjustable.

[0070] The line laser module provided in this embodiment, through the cooperation of the first camera 120 and the line laser emitting device 110, can identify obstacles or terrain in front of the main body of the device to perform corresponding obstacle avoidance or cleaning operations. The line laser emitting device 110 is movably connected to the main body 140, and / or the main body 140 is a movable structure. This allows adjustment of the azimuth and rotation angles of the line laser emitting device 110 by adjusting the position of the line laser emitting device 110 relative to the main body 140, and / or adjusting the relative positions of the components of the main body 140 itself. Furthermore, when the line laser emitting device 110 is assembled to the main body 140, it facilitates adjustment of the illumination angle and range of the line laser emitted by the line laser emitting device 110, enabling quick and convenient self-calibration of the line laser emitting device 110 and calibration in conjunction with the first camera 120. The operation is simple and convenient, improving assembly efficiency.

[0071] Furthermore, by reasonably adjusting the azimuth angle of the line laser emitting device 110, the line laser can be positioned within the field of view of the first camera 120. This ensures that the first camera 120 can accurately and comprehensively capture the reflected light emitted by the line laser emitting device 110 after being reflected by obstacles, thereby improving the accuracy and comprehensiveness of the environmental images acquired by the first camera 120. It is understood that by adjusting the azimuth angles of the line laser emitting devices 110 on both sides of the first camera 120, and then fixing the line lasers on both sides with adhesive, and by reasonably adjusting the rotation angle of the line laser emitting device 110 to make the line laser perpendicular to the horizontal plane, the ranging range can be improved.

[0072] Specifically, on the one hand, the line laser emitting device 110 is movably connected to the main body 140, so that the azimuth and rotation angle of the line laser emitting device 110 can be adjusted by adjusting the position of the line laser emitting device 110 relative to the main body 140; on the other hand, the main body 140 is a movable structure, so that the azimuth and rotation angle of the line laser emitting device 110 can be adjusted by adjusting the relative positions of its components; furthermore, the line laser emitting device 110 is movably connected to the main body 140, and the main body 140 is a movable structure, so that the azimuth and rotation angle of the line laser emitting device 110 can be adjusted by adjusting the position of the line laser emitting device 110 relative to the main body 140 and by adjusting the relative positions of its components. The embodiments provided in this disclosure adjust the azimuth and rotation angle of the line laser emitting device 110 in different ways, thus meeting the needs of different structures of the main body 140 and different connection methods between the line laser emitting device 110 and the main body 140.

[0073] Furthermore, the line laser emitting device 110 is used to emit a linearly projected line laser. For example, the line laser emitting device 110 emits a laser plane, which forms a line laser on the surface of an obstacle after reaching it. This line laser is used to detect environmental images. It is understood that a wave mirror can also be set in the emission direction (e.g., forward) of the line laser emitting device 110. Specifically, the wave mirror is a concave lens. For example, a concave lens is set in front of the laser tube. The laser tube emits light of a specific wavelength (e.g., infrared light), which becomes divergent light after passing through the concave lens, thus forming a straight line on the plane perpendicular to the optical path.

[0074] Line laser emitting devices 110 are distributed on both sides of the first camera 120. The first camera 120 and the line laser emitting devices 110 work together. That is, the first environmental image captured by the first camera 120 is the reflected light of the line laser emitted by the line laser emitting device 110 after being reflected by the obstacle. According to the first environmental image, the distance information of the obstacle can be obtained, and the distance between the obstacle and the main body of the device or the line laser module can also be measured, so as to perform corresponding obstacle avoidance operations.

[0075] The control system is housed on the main circuit board within the robot's body. It includes non-transitory memory such as hard drives, flash memory, and random access memory, and a computing processor such as a central processing unit (CPU) and application processor. The application processor uses obstacle information fed back by the laser rangefinder and localization algorithms, such as Simultaneous Localization and Mapping (SLAM), to create a real-time map of the robot's environment. Furthermore, it combines distance and speed information from sensors mounted on the buffer, cliff sensors, magnetometers, accelerometers, gyroscopes, and odometry to comprehensively determine the robot's current working state, location, and posture. This includes situations such as crossing thresholds, stepping onto carpets, being on a cliff, getting stuck above or below, a full dustbin, or being picked up. The system will then provide specific next action strategies for different situations, making the robot's operation more aligned with the user's requirements and providing a better user experience.

[0076] The drive system can manipulate the robot to move across the ground based on drive commands with distance and angle information (e.g., x, y, and θ components). The drive system includes drive wheel modules that can simultaneously control the left and right wheels. For more precise control of the robot's movement, it is preferable that the drive wheel modules each include a left drive wheel module and a right drive wheel module. The left and right drive wheel modules are opposed along a transverse axis defined by the main body. To enable the robot to move more stably or with greater mobility on the ground, the robot may include one or more driven wheels, including but not limited to omnidirectional wheels. The drive wheel module includes the walking wheels and drive motors, as well as control circuitry for controlling the drive motors. The drive wheel module may also be connected to circuitry for measuring drive current and an odometer. The drive wheel module can be detachably attached to the main body for easy disassembly and maintenance. The drive wheels may have an offset drop suspension system, movably secured, for example, rotatably attached, to the robot body and receiving spring biases that offset downwards and away from the robot body. Spring bias allows the drive wheel to maintain contact with the ground and traction with a certain ground force, while the cleaning element of the self-propelled device 100 also contacts the ground with a certain pressure.

[0077] The cleaning system can be a dry cleaning system or a wet cleaning system. In a dry cleaning system, the primary cleaning function originates from the sweeping system comprised of a roller brush, dustbin, fan, air outlet, and the connecting components between these four parts. The roller brush, with some interference from the ground, sweeps up debris from the ground and carries it to the suction port between the roller brush and the dustbin. There, it is drawn into the dustbin by the suction generated by the fan and passing through the dustbin. The dry cleaning system may also include a side brush 152 with a rotating shaft at an angle relative to the ground to move debris into the roller brush area of ​​the cleaning system.

[0078] The energy system includes rechargeable batteries, such as nickel-metal hydride (NiMH) and lithium-ion batteries. These batteries can be connected to a charging control circuit, a battery pack charging temperature detection circuit, and a battery undervoltage monitoring circuit. These circuits are then connected to a microcontroller control circuit. The main unit connects to a charging station via charging electrodes located on the side or bottom of the device. If dust adheres to the exposed charging electrodes, the accumulated charge during charging can cause the surrounding plastic casing to melt and deform, or even deform the electrodes themselves, preventing normal charging.

[0079] The human-computer interaction system includes buttons on the main control panel for users to select functions; it may also include a display screen and / or indicator lights and / or a speaker, which show the user the current status of the machine or available functions; and it may also include a mobile client application. For path navigation self-propelled devices, the mobile client can display a map of the environment in which the device is located, as well as the machine's position, and can provide users with richer and more user-friendly functions.

[0080] Figure 3 This is a flowchart illustrating an obstacle avoidance method for a self-propelled device based on a line laser, according to an exemplary embodiment.

[0081] like Figure 3 As shown, the obstacle avoidance method for self-propelled devices based on line lasers includes:

[0082] Step S301: Obtain the line laser recognition image of the self-propelled device.

[0083] Line laser recognition images can be obtained by illuminating the ground with a line laser.

[0084] Step S302: Determine whether there are wire-like obstacles based on the line laser recognition image to obtain the line laser recognition result;

[0085] In one embodiment, preferably, step S302 includes:

[0086] The line laser recognition image is analyzed to determine whether there are laser breakpoints in each line laser in the image;

[0087] When a laser breakpoint is present, it indicates the presence of a wire-type obstacle.

[0088] When there is no laser breakpoint, it is determined that there are no wire-type obstacles.

[0089] Within the obstacle avoidance distance range that the self-propelled device is concerned with, when there are no obstacles, the line laser illuminates the ground, and what is acquired is a set of continuous horizontal height (i.e., height is 0) ranging data, such as... Figure 4 The ground light, although rising due to hitting the wall in the distance, is not within the obstacle avoidance range that the self-propelled device is concerned with.

[0090] When a line laser's working range includes a line (a non-light-absorbing color or material), some of the light that should have illuminated the ground is blocked by the line itself. In this case, the ranging data can observe a small number of isolated ranging points at a certain height, such as... Figure 5 As shown, the arrows indicate discrete ranging points at a certain height, caused by light shining onto the cable. Simultaneously, light breaks are created at these points because the light is blocked by the cable itself.

[0091] like Figure 6 As shown, for some black or other light-absorbing materials, the emitted light cannot be observed on the wire itself. In this case, the characteristic of the ranging data is that the continuous horizontal height ranging data is truncated.

[0092] In this way, the distance to the wire can be determined by utilizing outlier ranging points or by the characteristic that the line laser is cut off when it encounters the ground, thus obtaining its location. Therefore, both wire identification and location information can be obtained simultaneously. Preferably, using the characteristic of being cut off, i.e., judging by the laser breakpoint, yields better results.

[0093] In some embodiments, in order to improve the accuracy of judgment, a comprehensive judgment can be made based on multiple laser breakpoints present in multiple / multiple line laser recognition results. When multiple laser breakpoints present in the multiple / multiple initial line laser recognition results form a continuous laser break with a length greater than a preset length, it is determined that there is a wire-type obstacle in these line laser recognition results.

[0094] Step S303: Based on the line laser recognition result, execute the corresponding obstacle avoidance action.

[0095] In this embodiment, the presence of wire-like obstacles can be determined by judging whether there are laser breakpoints in the line laser recognition image. If wire-like obstacles are present, the corresponding obstacle avoidance action is performed. If there are no wire-like obstacles, the self-propelled device walks normally without performing any obstacle avoidance operation.

[0096] In one embodiment, preferably, the method further includes:

[0097] Obtain image recognition results;

[0098] In response to the presence of a wire-type obstacle indicated by both the image recognition result and the line laser recognition result, the corresponding obstacle avoidance action is executed based on the image recognition result and the line laser recognition result.

[0099] In this embodiment, image recognition results can also be obtained to determine whether wire-type obstacles exist. Specifically, wire-type obstacles can be identified separately using the image recognition results.

[0100] However, since image recognition may make misjudgments and line laser ranging features may overfit (i.e., when some ground materials absorb / reflect light, the scene may be misjudged as potentially containing wires when the breakpoint is placed for judgment), in order to ensure the accuracy of the results, both image recognition results and line laser ranging results can be used simultaneously to identify wire obstacles when used as the basis for triggering obstacle avoidance behavior.

[0101] In one embodiment, preferably, based on the image recognition result and the line laser recognition result, a corresponding obstacle avoidance action is performed, including:

[0102] The first accuracy rate of historical image recognition results and the second accuracy rate of historical line laser recognition results are obtained respectively;

[0103] A first weight corresponding to the image recognition result is determined based on the first accuracy rate, and a second weight corresponding to the line laser recognition result is determined based on the second accuracy rate.

[0104] The comprehensive recognition result is determined based on the image recognition result, the first weight, the line laser recognition, and the second weight;

[0105] Based on the comprehensive recognition results, corresponding obstacle avoidance actions are performed on the wire-type obstacles.

[0106] In this embodiment, the weights of the image recognition results and the line laser recognition results can be determined based on their accuracy. For example, the recognition method that frequently makes mistakes and has low accuracy will have a lower weight value, while the recognition method with high accuracy will have a higher weight value, thereby ensuring higher accuracy of the recognition results.

[0107] In addition, when the presence of wire-type obstacles cannot be determined by a single image recognition result and a line laser recognition result, multiple recognition and verifications are required for each recognition method.

[0108] For recognition methods with high accuracy, fewer recognition attempts are needed to determine the presence of cables. Conversely, for methods with low accuracy, more recognition attempts are required to reach a conclusion about the presence of cables, and then a combined assessment of both methods is necessary. Alternatively, when one method has a sufficiently high accuracy, it can be relied upon primarily, while another method can be used as a simple auxiliary method or ignored altogether. For instance, when the image recognition results have low false negative and false positive rates, only a small number of ranging features are needed to determine the presence of cables, while when the false positive rate is high, more ranging features are required to determine the presence of cables.

[0109] Preferably, in response to the image recognition result indicating the presence of a wire-type obstacle and the line laser recognition result indicating the absence of a wire-type obstacle, or in response to the image recognition result indicating the absence of a wire-type obstacle and the line laser recognition result indicating the presence of a wire-type obstacle, the image recognition result and the line laser recognition result are acquired multiple times; the presence or absence of a wire-type obstacle is re-determined based on the multiple acquired image recognition result and line laser recognition result.

[0110] If only image recognition or line laser recognition results indicate the presence of a wire-like obstacle within a certain area, multiple image recognition or line laser recognition results are required. This multiple recognition essentially employs a probabilistic graph approach. Instead of making a judgment based on a single recognition result, the probability of an obstacle at a certain location is continuously adjusted through multiple recognitions. When an obstacle is recognized once, the probability of its presence in the corresponding area increases; when no target object is recognized in a frame, the probability of its presence within the robot's field of view decreases. After multiple recognitions, a probability distribution of obstacle presence in each area is formed. Then, based on preset probability thresholds, a final judgment is made, and the autonomous walking device performs corresponding obstacle avoidance or ignore behavior accordingly.

[0111] In this way, by detecting the above features in each frame of ranging data, it is possible to determine which parts of the entire known map area may have ranging features caused by scattered lines, thereby knowing the approximate distribution location of the scattered lines.

[0112] It should be noted that the presence of a wire obstacle can be determined by the line laser recognition result of a single location, and then the process can be repeated by turning or walking to finally obtain the distribution location of all wire obstacles in the environment; alternatively, the distribution location of all obstacles in the environment can be obtained by repeatedly using one recognition result, and then the distribution location of all obstacles in the environment can be obtained by repeatedly using another recognition result, and finally the two can be combined to obtain the final distribution location.

[0113] In one embodiment, preferably, the image recognition result includes a bounding box corresponding to the obstacle, and the method further includes:

[0114] Based on the coordinates of each laser breakpoint in the line laser recognition image, the first position coordinate information of the wire-type obstacle is determined;

[0115] The image recognition results are semantically segmented to determine whether each pixel in the location box corresponding to the wire-type obstacle belongs to the wire-type obstacle.

[0116] like Figure 7 As shown, in the image recognition results, for the target object, a minimum rectangle in the horizontal and vertical directions is given to determine the object range. However, for obstacles such as wires that are long and narrow and may have very irregular shapes, the effective part in the image is obviously too small, and most of them are relatively safe areas that do not require obstacle avoidance.

[0117] This invention, based on the already identified object bounding box, marks each pixel within the bounding box to indicate whether it belongs to the target object, such as... Figure 8 As shown, after this processing, the recognition result no longer simply provides a standard square image, but rather a more specific shape of the obstacle region, which helps to obtain the specific position coordinates of various parts of the wire. This improves the accuracy of wire position information in image recognition.

[0118] When a pixel belongs to the wire-type obstacle, the pixel is marked to determine the second position coordinate information of the wire-type obstacle;

[0119] Based on the first position coordinate information and the second position coordinate information, the final position coordinate information is determined;

[0120] In this embodiment, the union of the first position coordinate information and the second position coordinate information can be taken, and the union of the two can be determined as the final position coordinate information.

[0121] Based on the final position coordinates, execute the corresponding obstacle avoidance action.

[0122] Considering that self-propelled equipment is more likely to get the end of the wire caught in the side brush, roller brush rotating parts, etc., affecting subsequent operations, while the middle part of the wire is relatively safer, distinguishing between these two parts can better adjust the robot's response behavior.

[0123] In one embodiment, preferably, the method further includes:

[0124] Determine the start and end points of the wire-type obstacles;

[0125] A first obstacle avoidance action is performed in the first area where the start and end points of the wire-type obstacle are located, and a second obstacle avoidance action is performed in the second area other than the first area, wherein the avoidance distance of the first obstacle avoidance action is greater than the avoidance distance of the second obstacle avoidance action.

[0126] In this embodiment, after determining the start and end points of the wire-like obstacle, i.e. the wire end, obstacle avoidance can be triggered at a greater distance. For example, the non-wire end part is relatively safer, and a scheme to minimize missed scans is adopted, allowing the self-propelled device to approach a position at most A away from the wire. For the wire end part, in order to prevent the wire end from getting tangled, the self-propelled device is allowed to approach a position at most B away from the wire end, and B>A.

[0127] Specifically, by analyzing the direction of the wires in the image, it can be determined whether the observed area is a wire end. Specifically, wire ends can be identified in the following ways:

[0128] In one embodiment, preferably, determining the start and end points of the wire-like obstacle includes:

[0129] The image recognition result is identified using a target image recognition algorithm to determine the start and end points of the wire-type obstacle.

[0130] In this embodiment, the thread end can be directly treated as an object type for training and recognition, thereby identifying the specific range of the thread end.

[0131] In one embodiment, preferably, determining the start and end points of the wire-like obstacle includes:

[0132] Based on the first position coordinate information of the wire-type obstacle, determine the endpoint position of the wire-type obstacle;

[0133] The endpoint positions are determined as the start and end points of the wire-type obstacles.

[0134] In this embodiment, after identifying the entire cable through image recognition, the specific cable range, i.e., the first position coordinate information of cable-type obstacles, is obtained through semantic segmentation. Then, by tracking all pixels belonging to cable-type obstacles, the location of their continuous endpoints can be determined, for example, as... Figure 8As shown, it can be determined that within the frame, the endpoints of the wire are located in the upper left and lower right corners. These endpoints can then be considered the locations of the wire ends. Identifying the upper left corner as the wire end allows for increasing the obstacle avoidance distance in that area. While the lower right corner isn't a true wire end, it's outside the current blind spot and poses a risk of accidental entry and getting stuck; therefore, the obstacle avoidance distance in that area can also be increased. Additionally, a specific strategy can be implemented for false wire ends. For wire ends located at the edge of the overall image, their authenticity might be unclear. When approaching such a wire end, the current autonomous vehicle's behavior can be paused, the viewpoint turned towards the unknown wire end for observation, and the behavior resumed based on the updated identification results.

[0135] In one embodiment, preferably, determining the start and end points of the wire-like obstacle includes:

[0136] Obtain the multi-line laser recognition results of the self-propelled device, and determine all pixels within the range of the wire-type obstacle based on the multi-line laser recognition results;

[0137] Clustering is performed on all the pixels to determine the pixel breakpoints within the range of the wire-type obstacle.

[0138] The pixel breakpoints are determined as the start and end points of the wire-like obstacles.

[0139] In this embodiment, the results of multiple line laser scanning are used to obtain all range points that are identified as wires. By clustering, it is determined which ranges belong to 'one wire'. Then, by searching along the range of the wire, it can be determined where the wire breaks. The broken position can be considered as the wire end.

[0140] In this way, by continuously performing image recognition and segmentation processing on the self-propelled device during its movement, and in conjunction with the room map information established by the self-propelled device, it is possible to determine which locations within the entire known map area contain cables, and further identify which locations may be high-risk cable ends. Various behavioral strategies can then be implemented for different types of cable ends.

[0141] It should be noted that, based on the final image recognition result and the final line laser recognition result, it is not limited to using them to perform obstacle avoidance actions, as long as they can be used to identify the type and location of obstacles on the line. This information can be used for other purposes as needed.

[0142] Figure 9 This is a block diagram illustrating a line laser-based obstacle avoidance device for a self-propelled walking device, according to an exemplary embodiment.

[0143] like Figure 9As shown, according to a second aspect of the present disclosure, a self-propelled obstacle avoidance device based on line laser is provided, the device comprising:

[0144] Acquisition unit 91 is used to acquire the line laser recognition image of the self-propelled device;

[0145] The result determination unit 92 is used to determine whether there is a wire-like obstacle based on the line laser recognition image, so as to obtain the line laser recognition result;

[0146] The processing unit 93 is used to execute the corresponding obstacle avoidance action based on the line laser recognition result.

[0147] In one embodiment, preferably, the result determination unit is used for:

[0148] The line laser recognition image is analyzed to determine whether there are laser breakpoints in each line laser in the image;

[0149] When a laser breakpoint is present, it indicates the presence of a wire-type obstacle.

[0150] When there is no laser breakpoint, it is determined that there are no wire-type obstacles.

[0151] In one embodiment, preferably, the device further includes:

[0152] The result acquisition unit is used to acquire image recognition results;

[0153] The processing unit is configured to: respond to the presence of a wire-type obstacle indicated by both the image recognition result and the line laser recognition result, and perform a corresponding obstacle avoidance action based on the image recognition result and the line laser recognition result.

[0154] In one embodiment, preferably, the processing unit is used for:

[0155] The first accuracy rate of historical image recognition results and the second accuracy rate of historical line laser recognition results are obtained respectively;

[0156] A first weight corresponding to the image recognition result is determined based on the first accuracy rate, and a second weight corresponding to the line laser recognition result is determined based on the second accuracy rate.

[0157] The comprehensive recognition result is determined based on the image recognition result, the first weight, the line laser recognition, and the second weight;

[0158] Based on the comprehensive recognition results, corresponding obstacle avoidance actions are performed on the wire-type obstacles.

[0159] In one embodiment, preferably, the processing unit is further configured to:

[0160] In response to the image recognition result indicating the presence of a wire-type obstacle and the line laser recognition result indicating the absence of a wire-type obstacle, or in response to the image recognition result indicating the absence of a wire-type obstacle and the line laser recognition result indicating the presence of a wire-type obstacle, the image recognition result and the line laser recognition result are acquired multiple times.

[0161] Based on the image recognition results and line laser recognition results obtained from multiple acquisitions, it was re-determined whether there were any wire-type obstacles.

[0162] In one embodiment, preferably, the image recognition result includes a bounding box corresponding to the obstacle, and the device further includes:

[0163] The first coordinate determination unit is used to determine the first position coordinate information of the wire-like obstacle based on the coordinates of each laser breakpoint in the line laser recognition image;

[0164] A pixel determination unit is used to perform semantic segmentation on the image recognition result and determine whether each pixel in the location box corresponding to the wire obstacle belongs to the wire obstacle.

[0165] The annotation unit is used to annotate the pixel when the pixel belongs to the wire-type obstacle, so as to determine the second position coordinate information of the wire-type obstacle;

[0166] The second coordinate determination unit is used to determine the final position coordinate information based on the first position coordinate information and the second position coordinate information;

[0167] The processing unit is also used to: execute corresponding obstacle avoidance actions based on the final position coordinate information.

[0168] In one embodiment, preferably, the device further includes:

[0169] A start and end point determination unit is used to determine the start and end points of the wire-like obstacles;

[0170] The processing unit is used to: perform a first obstacle avoidance action in a first area where the start and end points of the wire-type obstacle are located, and perform a second obstacle avoidance action in a second area other than the first area, wherein the avoidance distance of the first obstacle avoidance action is greater than the avoidance distance of the second obstacle avoidance action.

[0171] In one embodiment, preferably, the start and end point determination unit is used for:

[0172] The image recognition result is identified using a target image recognition algorithm to determine the start and end points of the wire-type obstacle.

[0173] In one embodiment, preferably, the start and end point determination unit is used for:

[0174] Based on the first position coordinate information of the wire-type obstacle, determine the endpoint position of the wire-type obstacle;

[0175] The endpoint positions are determined as the start and end points of the wire-type obstacles.

[0176] In one embodiment, preferably, the start and end point determination unit is used for:

[0177] Obtain the multi-line laser recognition results of the self-propelled device, and determine all pixels within the range of the wire-type obstacle based on the multi-line laser recognition results;

[0178] Clustering is performed on all the pixels to determine the pixel breakpoints within the range of the wire-type obstacle.

[0179] The pixel breakpoints are determined as the start and end points of the wire-like obstacles.

[0180] According to a third aspect of the present disclosure, a self-propelled device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor.

[0181] The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of the first aspect embodiments above.

[0182] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in any of the embodiments of the first aspect.

[0183] It can be further understood that in this disclosure, "multiple" refers to two or more, and other quantifiers are similar. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. The singular forms "a," "the," and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

[0184] It is further understood that the terms "first," "second," etc., are used to describe various types of information, but this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another, and do not indicate a specific order or degree of importance. In fact, the expressions "first," "second," etc., are completely interchangeable. For example, without departing from the scope of this disclosure, first information can also be referred to as second information, and similarly, second information can also be referred to as first information.

[0185] It is further understood that although operations are described in a specific order in the accompanying drawings in the embodiments of this disclosure, this should not be construed as requiring these operations to be performed in the specific order or serial order shown, or requiring all of the shown operations to be performed to obtain the desired result. In certain environments, multitasking and parallel processing may be advantageous.

[0186] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0187] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for obstacle avoidance in a self-propelled device based on line laser, characterized in that, The method includes: Acquire the line laser recognition image of the self-propelled device; The line laser recognition image is analyzed to determine whether there are laser breaks in each line laser in the image, and the presence or absence of laser breaks determines whether there are wire-like obstacles, so as to obtain the line laser recognition result. Based on the line laser recognition results, the corresponding obstacle avoidance action is executed.

2. The obstacle avoidance method for self-propelled devices according to claim 1, characterized in that, The presence of wire-type obstacles can be determined by the presence of laser breakpoints, including: When a laser breakpoint is present, it indicates the presence of a wire-type obstacle. When there is no laser breakpoint, it is determined that there are no wire-type obstacles.

3. The obstacle avoidance method for self-propelled devices according to claim 2, characterized in that, The method further includes: Obtain image recognition results; In response to the presence of a wire-type obstacle indicated by both the image recognition result and the line laser recognition result, the corresponding obstacle avoidance action is executed based on the image recognition result and the line laser recognition result.

4. The obstacle avoidance method for self-propelled devices according to claim 3, characterized in that, Based on the image recognition result and the line laser recognition result, perform corresponding obstacle avoidance actions, including: The first accuracy rate of historical image recognition results and the second accuracy rate of historical line laser recognition results are obtained respectively; A first weight corresponding to the image recognition result is determined based on the first accuracy rate, and a second weight corresponding to the line laser recognition result is determined based on the second accuracy rate. The comprehensive recognition result is determined based on the image recognition result, the first weight, the line laser recognition, and the second weight; Based on the comprehensive recognition results, corresponding obstacle avoidance actions are performed on the wire-type obstacles.

5. The obstacle avoidance method for self-propelled devices according to claim 3, characterized in that, The method further includes: In response to the image recognition result indicating the presence of a wire-type obstacle and the line laser recognition result indicating the absence of a wire-type obstacle, or in response to the image recognition result indicating the absence of a wire-type obstacle and the line laser recognition result indicating the presence of a wire-type obstacle, the image recognition result and the line laser recognition result are acquired multiple times. Based on the image recognition results and line laser recognition results obtained from multiple acquisitions, it was re-determined whether there were any wire-type obstacles.

6. The obstacle avoidance method for self-propelled devices according to claim 3, characterized in that, The image recognition result includes the bounding box corresponding to the obstacle, and the method further includes: Based on the coordinates of each laser breakpoint in the line laser recognition image, the first position coordinate information of the wire-type obstacle is determined; The image recognition results are semantically segmented to determine whether each pixel in the location box corresponding to the wire-type obstacle belongs to the wire-type obstacle. When a pixel belongs to the wire-type obstacle, the pixel is marked to determine the second position coordinate information of the wire-type obstacle; Based on the first position coordinate information and the second position coordinate information, the final position coordinate information is determined; Based on the final position coordinates, execute the corresponding obstacle avoidance action.

7. The obstacle avoidance method for self-propelled devices according to claim 3, characterized in that, The method further includes: Determine the start and end points of the wire-type obstacles; A first obstacle avoidance action is performed in the first area where the start and end points of the wire-type obstacle are located, and a second obstacle avoidance action is performed in the second area other than the first area, wherein the avoidance distance of the first obstacle avoidance action is greater than the avoidance distance of the second obstacle avoidance action.

8. The obstacle avoidance method for self-propelled devices according to claim 7, characterized in that, Determining the start and end points of the wire-type obstacles includes: The image recognition result is identified using a target image recognition algorithm to determine the start and end points of the wire-type obstacle.

9. The obstacle avoidance method for self-propelled devices according to claim 7, characterized in that, Determining the start and end points of the wire-type obstacles includes: Based on the first position coordinate information of the wire-type obstacle, determine the endpoint position of the wire-type obstacle; The endpoint positions are determined as the start and end points of the wire-type obstacles.

10. The obstacle avoidance method for self-propelled devices according to claim 7, characterized in that, Determining the start and end points of the wire-type obstacles includes: Obtain the multi-line laser recognition results of the self-propelled device, and determine all pixels within the range of the wire-type obstacle based on the multi-line laser recognition results; Clustering is performed on all the pixels to determine the pixel breakpoints within the range of the wire-type obstacle. The pixel breakpoints are determined as the start and end points of the wire-like obstacles.

11. An obstacle avoidance device for self-propelled equipment based on line laser, characterized in that, The device includes: The acquisition unit is used to acquire the line laser recognition image of the self-propelled device; The result determination unit is used to analyze the line laser recognition image, determine whether there are laser breaks in each line laser in the line laser recognition image, and determine whether there are wire-like obstacles based on whether there are laser breaks, so as to obtain the line laser recognition result. The processing unit is used to execute the corresponding obstacle avoidance action based on the line laser recognition result.

12. A self-propelled device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The device stores computer-executable instructions for performing the method flow as described in any one of claims 1 to 10.

Images