Control method of self-moving device, self-moving device and storage medium

Abstract

The invention discloses a control method of a self-moving device, the self-moving device and a storage medium, and the method comprises the steps: obtaining the region type of each region in a collected image, each region at least comprises a to-be-driven region and a target region, the to-be-driven region is a part of a working region, and the target region is a part of the to-be-driven region; the target area is an area adjacent to the to-be-driven area; and when the area type of the target area is a first area type, controlling the self-moving equipment to change a driving mode, so that the coverage range of the working part comprises a part of the target area. Thus, the self-moving device can process the boundary of the working area on the premise of considering the safety, the processing coverage rate of the boundary of the working area is improved, and the user experience is improved.

Description

Self-moving device control method, self-moving device and storage medium Technical Field

[0001] The present invention relates to the field of intelligent control technology, and in particular to a control method for a self-moving device, a self-moving device, and a storage medium. Background Art

[0002] As the development trend of intelligence is getting faster and faster, autonomous mobile devices have been widely used in various industries and people's daily lives. Autonomous mobile devices can walk according to pre-set paths or areas and perform related tasks.

[0003] Commonly used autonomous devices include automatic lawn mowers and automatic sprinklers. Taking an automatic lawn mower as an example, since there may be objects outside the boundary of the working area that cannot be cut, such as walls, roads, flowers, etc., for safety reasons, automatic lawn mowers usually only cut inside the boundary. In this case, the cutting range of the body shell and the blade disc are both located inside the boundary, or there is a certain distance from the inside of the boundary, resulting in the grass at the boundary not being cut cleanly.

[0004] Summary of the Invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides a control method for a self-moving device, a self-moving device and a storage medium.

[0006] According to a first aspect of an embodiment of the present disclosure, a method for controlling a self-moving device is provided. The self-moving device includes a working portion configured to perform a work operation, and the self-moving device further includes an image acquisition device configured to acquire an image in front of the self-moving device. The self-moving device is configured to travel and / or work in a work area. The method includes:

[0007] Acquire a region type of each region in the acquired image, wherein each region includes at least: a to-be-traveled region and a target region, wherein the to-be-traveled region is a portion of the working region, and the target region is a region adjacent to the to-be-traveled region;

[0008] When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working unit includes a portion of the target area.

[0009] In one embodiment, the method further includes: inputting the collected image into a trained image recognition model to perform semantic segmentation on the image to obtain the region type of each region.

[0010] In one embodiment, the method further comprises:

[0011] When the working mode is a preset working mode, when the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working part includes a part of the target area. The preset working mode is an edge working mode or a working mode preset by the user. The edge working mode is a mode for controlling the self-moving device to move and work along the boundary of the working area.

[0012] In one embodiment, the first area type includes one or more of roads, manhole covers, stone roads, hard roads, fallen leaves and mud, or the first area type is obtained according to a user instruction.

[0013] In one embodiment, the method further includes: when a contact collision is detected during the process in which the working portion covers a portion of the target area, controlling the self-moving device to change a driving path so as to drive toward the inside of the working area.

[0014] In one embodiment, obtaining the region type of each region in the acquired image further includes:

[0015] performing depth estimation processing on the image to determine depth information corresponding to the target area, the depth information being used to represent the distance between each position of the target area and the image acquisition device;

[0016] Accordingly, when the area type of the target area is the first area type, controlling the self-moving device to change the driving mode includes:

[0017] The self-moving device is controlled to change its driving mode according to the area type of the target area and the depth information, so that when the area type of the target area is the first area type and the height difference between the target area and the area to be driven is less than a height threshold, the coverage range of the working unit includes a part of the target area.

[0018] In one embodiment, semantic segmentation is performed on the image to obtain the region type of each region. The region type of the to-be-traveled region is a grassland type, and the region type of the target region is a first region type. Accordingly,

[0019] Controlling the coverage of the working portion to include a portion of the target area comprises:

[0020] Marking the area type of the portion of the target area adjacent to the area to be driven as a grassland type;

[0021] The self-moving device is controlled to travel and / or operate along a boundary between an area defined by the grass type and an area defined by the first area type.

[0022] In one embodiment, when the area type of the target area is the second area type, the self-moving device is controlled to travel and / or work within the working area.

[0023] In one embodiment, the second area type includes one or more of a wall, a pit, a pool, a fence, a flower bed, soil with a hardness less than a hardness threshold, an irrigation facility area, and a movable living body area.

[0024] In one embodiment, controlling the self-moving device to travel and / or work in the working area includes:

[0025] Get edge distance information;

[0026] The self-moving device is controlled to travel and / or work in the working area according to the edge distance information.

[0027] In one embodiment, obtaining edge distance information includes:

[0028] Get user command information;

[0029] The edge distance information is determined according to the user instruction information.

[0030] In one embodiment, the method further includes: when the area type of the target area is a first area type, controlling the self-moving device to change the driving mode so that the self-moving device avoids the area where the second area type is located, wherein the area type of the target area includes a first area type and a second area type, and the area where the first area type is located and the area where the second area type is located are both adjacent to the area to be driven.

[0031] In one embodiment, the method further includes: when the area type of the target area is the first area type, controlling the self-moving device to change the driving mode so that the driving distance of the working unit is less than the boundary length between the area where the first area type is located and the area to be driven, wherein the driving distance is the distance traveled by the self-moving device when the coverage range of the working unit includes a part of the target area.

[0032] In one embodiment, before obtaining the area type of each area in the captured image, the self-moving device travels along a first path; when the area type of the target area is the first area type, controlling the self-moving device to change the driving mode includes:

[0033] The self-moving device is controlled to continue traveling along the first path for a period of time, and after traveling for a period of time, the traveling path is changed so that the coverage range of the working part includes a part of the target area.

[0034] According to a second aspect of the embodiments of the present disclosure, the embodiments of the present application provide a self-moving device, the self-moving device comprising:

[0035] a working unit configured to perform a work operation;

[0036] an image acquisition device, the image acquisition device being configured to acquire an image in front of the mobile device;

[0037] A controller is signal-connected to the working portion and the image acquisition device, and is configured to control the self-moving device to travel and / or work in a working area, including:

[0038] The controller obtains the area type of each area in the acquired image, wherein each area includes at least: a to-be-traveled area and a target area, wherein the to-be-traveled area is a part of the working area, and the target area is an area adjacent to the to-be-traveled area;

[0039] When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working unit includes a portion of the target area.

[0040] According to a third aspect of the embodiments of the present disclosure, the embodiments of the present application provide a computer-readable storage medium, in which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0041] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: obtaining the area type of each area in the captured image, and when the area type of the target area is the first area type, controlling the self-moving device to change the driving mode so that the coverage of the working part includes a part of the target area. The self-moving device can process the boundaries of the working area while taking safety into consideration, thereby improving the processing coverage rate of the boundaries of the working area and enhancing the user experience. Furthermore, the self-moving device can quickly identify the area type of each area in the image, that is, it can quickly identify multiple scenes, so that different working strategies can be selected for different scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] FIG1 is a schematic flow chart of a method for controlling a mobile device according to an embodiment of the present invention;

[0043] FIG2 is a first schematic diagram of a working area according to an embodiment of the present invention;

[0044] FIG3 is a second schematic diagram of a working area according to an embodiment of the present invention;

[0045] FIG4 is a third schematic diagram of a working area according to an embodiment of the present invention;

[0046] FIG5 is a schematic diagram of the offset between each boundary of the working area and the charging station in an embodiment of the present invention;

[0047] FIG6 is a schematic diagram of a scenario in which an obstacle area is detected by a mobile device according to an embodiment of the present invention;

[0048] FIG7 is a first image captured by a mobile device according to an embodiment of the present invention;

[0049] FIG8 is a second image provided by an embodiment of the present invention after marking the area type of the portion of the first image adjacent to the area to be driven as grass type;

[0050] FIG9 is a schematic diagram of a scenario in which a mobile device is traveling along a boundary of a working area within a working area according to an embodiment of the present invention;

[0051] FIG10 is a schematic diagram of a scenario in which a mobile device performs a cross-edge action according to an embodiment of the present invention;

[0052] 11-12 are schematic diagrams of another scenario in which a mobile device performs a cross-border action according to an embodiment of the present invention;

[0053] 13-14 are schematic diagrams of another scenario in which a mobile device performs a cross-border action according to an embodiment of the present invention;

[0054] FIG15 is a schematic diagram of the structure of a self-moving device provided in an embodiment of the present invention;

[0055] FIG16 is a schematic diagram of a working module of a self-moving device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0057] It should be noted that, in this document, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, components, features, and elements with the same name in different embodiments of the present application may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context of the specific embodiment.

[0058] It should be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are merely used to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the term "if" as used herein may be interpreted as "at the time of," "when," or "in response to a determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms "comprising" and "including" indicate the presence of the described features, steps, operations, elements, components, items, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, types, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, meaning any one or any combination. Thus, “A, B, or C” or “A, B, and / or C” means “any of: A; B; C; A and B; A and C; B and C; A, B, and C.” An exception to this definition occurs only when a combination of elements, functions, steps, or operations are inherently mutually exclusive in some manner.

[0059] It should be understood that, although the various steps in the flowchart in the embodiment of the present application are shown in sequence according to the indication of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and they can be performed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and their execution order is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0060] It should be noted that in this article, step codes such as S101 and S102 are used for the purpose of expressing the corresponding content more clearly and concisely, and do not constitute a substantial limitation on the order. When implementing the step, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the scope of protection of this application.

[0061] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0062] In the subsequent description, the use of suffixes such as "module", "component" or "unit" to represent elements is only for the purpose of facilitating the description of the present application and has no specific meaning. Therefore, "module", "component" or "unit" can be used interchangeably.

[0063] Considering that currently, when autonomous mobile devices operate within a work area defined by boundaries, their coverage at the boundaries of the work area is low. For example, an autonomous lawn mower, when cutting, has low coverage at the boundaries of its work area. This is primarily due to three factors: First, the outer edge of the boundary may be flowers planted by the user or obstacles such as fences, walls, and roads. To avoid cutting ornamental flowers or colliding with obstacles, autonomous lawn mowers typically cut within the boundary, with the entire machine (including the housing and the cutting area of ​​the blade disc) positioned within the boundary, sometimes even at a distance from it. Second, vision-based autonomous lawn mowers rely on visual cameras for movement. In scenarios where cameras identify boundaries and move along them, the lateral distance between the boundary and the machine body, determined in real time through vision, is often inaccurate, resulting in a gap between the machine body and the boundary. Furthermore, for safety reasons, the blade disc's cutting area in autonomous lawn mowers typically does not extend beyond the machine body, potentially leading to missed areas between the blade disc and the boundary. At least due to the above three reasons, after the machine completes the cutting work in the working area, it will still identify the presence of uncut grass in the working area, and most of the uncut grass is concentrated near the boundary.

[0064] However, users may have the following requirements: they want a clean cut at the boundary between a road and a lawn; or they want a clean cut at the boundary between a muddy field and a lawn; or, because the visual recognition model only classifies images into two categories, grass and non-grass, a pile of yellow fallen leaves on the lawn may be identified as non-grass, with the boundary between grass and non-grass forming the boundary. This prevents the automatic lawn mower from cleanly cutting around the fallen leaves, despite the user's actual desire for a clean cut. In other words, there is a need to address the problem of how existing autonomous vehicles can handle grass at the boundaries of their work areas while ensuring safety. Furthermore, in actual applications, there are a variety of scenarios, such as roads, muddy fields, fences, and walls, and users may have different requirements for clean cuts in different scenarios. In other words, it is also necessary for the autonomous vehicle to be able to quickly identify a variety of scenarios and adopt different operating strategies for different scenarios. Based on this, as shown in Figure 1, this application proposes a control method for an autonomous vehicle. It should be noted that while this disclosure provides the method operating steps shown in the following embodiments or accompanying figures, the method may include more or fewer operating steps based on routine or uninventive effort. For steps that do not have a necessary causal relationship logically, the execution order of these steps is not limited to the execution order provided in the embodiments of the present disclosure.

[0065] Referring to FIG. 1 , a control method for a self-propelled device provided in an embodiment of the present application is shown. For example, the self-propelled device includes a working portion and is configured to travel and / or work in a working area. The method provided in this embodiment includes:

[0066] The area type of each area is obtained, where each area at least includes: an area to be driven and a target area, wherein the target area is an area adjacent to the working area.

[0067] This embodiment provides a self-moving device, as shown in Figures 1, 15 and 16. The self-moving device includes a body 27, an imaging sensor 200 (hereinafter also referred to as a visual sensor, or an image acquisition device 28), a position sensor 500 and a control circuit 600.

[0068] Specifically, the body 27 includes a drive device 700, which is used to drive the body 27 on the work surface according to received drive commands. The drive device 700 generally includes rollers and a motor that drives the rollers. The rollers may include a driving wheel and a driven wheel. The rollers 211 and 212 may be located on both sides of the body 27, with one or two rollers on each side.

[0069] The body 27 also includes a working unit, which is used to perform specific work tasks. For example, if the self-moving device is an automatic lawn mower, the working unit includes a mowing disc 221, a cutting motor 222, and other components. It may also include auxiliary components such as a mowing height adjustment mechanism to optimize or adjust the mowing effect. For example, if the self-moving device is an automatic vacuum cleaner, the working unit includes a vacuum motor, a vacuum port, a vacuum tube, a vacuum chamber, a dust collection device, and other working components used to perform the vacuuming task.

[0070] The body 27 may also include an energy module, which is used to provide energy for various operations of the self-mobile device. The energy module may include a rechargeable battery and a charging connection structure, wherein the charging connection structure is usually a charging electrode sheet, which can be used in conjunction with a charging electrode sheet set at the docking station to charge the self-mobile device.

[0071] The body 27 further includes a memory 400, which is used to store data generated by sensors or control circuits, or to pre-store data for use by the control circuits.

[0072] The fuselage 27 also includes a position sensor 500 , which may include a satellite positioning sensor, an inertial sensor IMU, or an odometer ODO installed on the driving device 700 , etc., and is used to obtain a relative position according to the movement of the fuselage 27 .

[0073] In addition to the above modules, the body 27 may also include a shell for accommodating and installing each module, a control panel for user operation, etc. It may also include various environmental sensors, such as humidity sensors, temperature sensors, acceleration sensors, light sensors, etc. The above sensors can assist the mobile device in judging the working environment to execute the corresponding program.

[0074] The control circuit 600 (also known as the controller) is the core component of the self-moving device, which is used to control the automatic movement and operation of the self-moving device. The functions it performs include controlling the working module to start or stop working, generating a moving path and controlling the drive device 700 to move according to the path, judging the power level of the energy module and promptly controlling the self-moving device to return to the docking station for automatic docking and charging, and executing corresponding programs based on the data from the environmental sensor.

[0075] Referring to Figures 1, 15, and 16, the self-propelled device includes an imaging sensor 200. Imaging sensor 200 is connected to body 100 and is used to capture images in the forward direction of body 27. These images, at least in part, capture images of the working surface in the forward direction. The captured images are within a field of view 210 of imaging sensor 200. Imaging sensor 200 can be a camera or laser radar commonly used in the industry.

[0076] Typically, the imaging sensor 200 is mounted near the front upper portion of the fuselage 27, preferably centered, with its viewing angle facing forward and downward to capture images of the work surface. The size of its field of view can be adjusted based on actual needs. A larger field of view 210 captures more images in the forward direction of the fuselage 27, and vice versa. The fuselage 27 can have various forward directions, such as normal forward movement, backward movement, and turning movements. In this embodiment, the forward direction of the fuselage refers to the normal forward direction, i.e., the direction of the fuselage's central axis. In this application, the imaging sensor 200 can be a camera.

[0077] In this embodiment, the working area can be the area where the self-moving device travels and / or works, and the working area is defined by a boundary. Of course, the self-moving device can also travel outside the working area to perform work. The area to be traveled can be a part of the working area, for example: it can be the working area within a specified range in front of the moving direction of the self-moving device, or the area marked on the working area where the self-moving device is located. For example, taking the self-moving device as an automatic lawn mower as an example, the working area can be a grass area in front of the moving direction of the self-moving device and within 3 meters of the self-moving device. It can be understood that the area to be traveled can be a partial area in the working area, that is, any area in the working area, and the target area can be any area outside the working area and adjacent to the area to be traveled.

[0078] It is understood that there may be only one or more areas adjacent to the area to be driven, that is, there may be one or more target areas. It should be noted that the area to be driven and the target area may both be within the working area, or only the area to be driven may be within the working area, while the target area is not.

[0079] Among them, the area type can be used to indicate the attributes of the area. Optionally, the area type can be the name of the area, such as road, stone road, hard road surface, manhole cover, earth pit, feces, fallen leaves, mud, wall, tree, fence, flower bed, stone, step, pool, soft soil, irrigation facility area, movable life area, etc.; wherein the soil hardness of soft soil is less than the hardness threshold, and the soil hardness of mud is higher than that of soft soil; loose soil, such as granular soil, may cause a self-moving device to sink and become trapped when driving on the soft soil; the irrigation facility area can be the area where the irrigation facilities (such as faucets) are located; the movable life can be a person or an animal, and the movable life can move on its own; the movable life area is the area where the movable life is located.

[0080] In one possible implementation, the self-moving device includes an image acquisition device configured to capture images in front of the self-moving device, the images including images corresponding to the to-be-traveled area and images corresponding to the target area. The images are processed to determine the area type of each area in the image.

[0081] Specifically, the self-moving device may trigger the image acquisition device to capture images in real time, sporadically, or periodically, to obtain images corresponding to the area to be driven and the target area. The captured images are then processed in a predetermined manner to determine the area type of the target area. It is understood that the self-moving device may also trigger the image acquisition device to capture images only when it is near the boundary of the area to be driven, thereby saving energy. The self-moving device may determine whether it is near the boundary of the area to be driven based on the relationship between its own position and the position of the boundary of the area to be driven.

[0082] The image acquisition device may be specifically a camera or other device, and the image acquisition range corresponding to the image acquisition device may be set according to actual needs, and may generally include a specified range in front of and on both sides of the moving direction of the self-moving device. Taking the self-moving device as an automatic lawn mower as an example, as shown in FIG2 , the dotted box in FIG2 represents the image acquisition range corresponding to the image acquisition device of the automatic lawn mower. When the automatic lawn mower moves close to the boundary of the lawn, the acquired images include both images corresponding to the lawn area and images corresponding to non-grass areas, such as images corresponding to a pool or a wall. Step S101 may be performed according to actual needs: obtaining the area type of each area in the image, each area including at least: a to-be-traveled area and a target area, wherein the to-be-traveled area is part of the working area, and the target area is an area adjacent to the to-be-traveled area. The method of obtaining the type includes, but is not limited to, semantic segmentation processing, classification processing, etc., and is not specifically limited here.

[0083] In the embodiment of the present disclosure, an image acquisition device is used to collect images corresponding to the area to be driven and the target area from a mobile device, and the images are pre-processed to obtain the area type of the target area. The operation is convenient and the accuracy is high.

[0084] In one possible implementation, the image may include a two-dimensional image, and the collected image is input into an image recognition model trained in advance, and each pixel in the image is classified to determine the category of each pixel (such as grass, cobblestone road or fence, etc.), thereby achieving the goal of semantic segmentation of the image (semantic segmentation is also referred to as image segmentation below) to identify the region type of each region. The image recognition model can be a multi-classification model, that is, it can identify more than two region types, such as whether the region type is grass, road, or wall. It can be understood that the region classification model established based on the classification algorithm can be trained in advance with training samples, and the training samples may include image samples of different regions and corresponding region type labels. For example, the image recognition model can be a neural network model. By performing semantic segmentation on the image, compared with other methods such as depth information that only identifies scenes with obstacles, the present application can quickly identify a variety of richer scenes such as roads, mud, fences, walls, etc., so that subsequent users can set different cutting requirements for different scenes.

[0085] In one embodiment of the present application, semantic segmentation of the image results in the area type of the area to be driven being a grassland type, and the type of the target driving area being a first area type or a second area type, which may also be collectively referred to as a target type. As described above, the area type may be used to indicate the attributes of an area, and the grassland area in the image represents the area where the grassland is spatially mapped onto the image. The attributes or area types of the grassland area may be referred to as grassland attributes, grassland types, or simply as grassland. The target area in the image represents the area where the target object is spatially mapped onto the image, and the attributes or area types of the target area are target types, which may also be the first area type or the second area type.

[0086] Taking the mobile device as an automatic lawn mower as an example, referring to FIG2 , after obtaining an image corresponding to any virtual frame, the automatic lawn mower performs semantic segmentation processing on the image to determine the presence of walls, pools, fallen leaves, etc. in the image.

[0087] Exemplarily, the user can also set identification devices (such as identification signs, identification lines, etc.) corresponding to the area type in each area. The mobile device can detect and identify the identification device based on the image, determine the identification information corresponding to the identification device, and identify the area type of the target image based on the identification information.

[0088] In the embodiment of the present disclosure, a mobile device performs semantic segmentation processing on an image corresponding to the area to be driven and an image corresponding to the target area to obtain the corresponding area type. No user participation in the setting is required, and the processing speed is fast.

[0089] In one possible implementation, the image includes a depth image, and the image is processed to determine the area type of each area in the image, including: performing depth estimation processing on the image to determine the depth information corresponding to the target area, the depth information is used to represent the distance between each position of the target area and the self-mobile device; based on the depth information, determine the area type of the target area.

[0090] Specifically, depth estimation processing is performed on the depth image including the image corresponding to the area to be driven and the image corresponding to the target area to determine the depth information corresponding to the target area, and the depth information is used to represent the distance between each position of the target area and the self-moving device. Then, the area type of the target area is determined based on the change amplitude information of the depth information.

[0091] Among them, performing depth estimation processing on an image refers to estimating the distance between the object represented by each pixel in the image and the self-mobile device, which can be specifically processed by a depth estimation network model, etc. Since different objects (i.e., areas) are photographed at the same time, the pixels of different objects in the depth image have different change characteristics. For example, the farther away from the image acquisition device of the self-mobile device, the adjacent pixels of normal grass will gradually increase, the adjacent pixels of the wall will not change, and the adjacent pixels of the depression will increase sharply. Therefore, the area type of the target area can be determined based on the change amplitude information of the depth information corresponding to the target area. It should be noted that the specific process of performing depth estimation processing on an image can refer to the existing technology and will not be repeated here. In addition, depth information includes pixel information.

[0092] In one possible implementation, the region type of the target region is determined based on the change amplitude information of the depth information, including: when the change amplitude information represents that the depth information is continuously changing, determining the region type of the target region as the first region type; and / or, when the change amplitude information represents that the depth information is discontinuously changing, determining the region type of the target region as the second region type.

[0093] It can be understood that since the change amplitude information of the depth information corresponding to different areas has different characteristics, for example, adjacent pixels in areas such as flat land will change continuously, while adjacent pixels in areas such as concave land and walls will change discontinuously. At the same time, areas where adjacent pixels change continuously can be considered safe for self-mobile equipment, while areas where adjacent pixels change discontinuously can be considered dangerous for self-mobile equipment. Therefore, the area type of the target area can be determined based on the change amplitude information of the depth information, that is, when the change amplitude information represents that the depth information is continuously changing, the area type of the target area is determined to be the first area type, and / or, when the change amplitude information represents that the depth information is discontinuously changing, the area type of the target area is determined to be the second area type.

[0094] In one possible implementation, obtaining the area type of each area includes: detecting a boundary signal and determining a detection result, where the boundary signal includes a magnetic field signal generated by a boundary line set at the boundary of the area to be driven; and determining the area type of each area based on the detection result.

[0095] It can be understood that whether to set a boundary line on the target boundary can be determined in advance based on the area type of the target area adjacent to the area to be traveled, so as to generate a boundary signal through the boundary line. For example, when the target area adjacent to the area to be traveled is a dangerous area such as a pool or a wall, a boundary line can be set on the target boundary between the area to be traveled and the target area, so as to generate a boundary signal for indicating that the target area is a dangerous area through the boundary line; and when the target area adjacent to the area to be traveled is a safe area such as soil or fallen leaves, a boundary line may not be set on the target boundary between the area to be traveled and the target area, so as not to generate a boundary signal.

[0096] Here, a boundary line is a conductor that forms a loop when energized. When a constant current is passed through the boundary line, a constant magnetic field surrounding the boundary line is generated. During operation, the autonomous mobile device can detect the magnetic field signal of this constant magnetic field, i.e., the boundary signal. Specifically, the boundary line can be a geomagnetic line, for example, and the geomagnetic line can be a permanent magnet, a current-carrying conductor, or the like.

[0097] Among them, the self-moving device may be provided with a signal detection device for detecting boundary signals. The self-moving device may detect boundary signals in real time, irregularly or periodically and determine the detection results, and then determine the area type of the target area based on the detection results. For example, if the detection result shows that no boundary signal is detected, it means that the area type of the target area or the boundary type of the target boundary is safe; if the detection result shows that a boundary signal is detected, it means that the area type of the target area or the boundary type of the target boundary is dangerous, etc. It is understandable that the self-moving device may also trigger the detection boundary signal only when it is close to the boundary of the area to be traveled, in order to save energy consumption, and the self-moving device may determine whether it is close to the boundary of the area to be traveled based on its own position and the position of the boundary of the area to be traveled.

[0098] In one possible implementation, determining the area type of the target area or the boundary type of the target boundary based on the detection results includes: if no boundary signal is detected, determining the area type of the target area to be the first area type; if a boundary signal is detected, determining the area type of the target area to be the second area type, or determining the boundary type of the target boundary to be the second boundary type. Specifically, if the detection results determine that no boundary signal is detected, it indicates that it is safe for the self-moving device or working unit to move and operate within the target area or at the target boundary, and the area type of the target area is determined to be the first area type. For example, if the area to be traveled is grassland and the target area is fallen leaves, a boundary line may not be set at the boundary between the grassland and fallen leaves, so that the boundary signal cannot be detected when the self-moving device moves near the boundary between the grassland and fallen leaves. If the detection results determine that a boundary signal is detected, it indicates that it is dangerous for the self-moving device or working unit to move and operate within the target area or at the target boundary, and the area type of the target area is determined to be the second area type. For example, if the area to be driven is a grassland and the target area is a pond, a boundary line can be set at the boundary between the grassland and the pond, so that a boundary signal is detected when the mobile device moves near the boundary between the grassland and the pond.

[0099] Taking an autonomous lawn mower as an example (see Figure 3), if the mower moves over a puddle, it could fall, and if it moves near a wall, it could damage the cutterhead. Puddles and walls are considered dangerous areas. Therefore, geomagnetic lines are set at the boundaries between the grass and the puddle, or the wall. When the mower approaches the boundary between the grass and the puddle, or the boundary between the grass and the wall, a boundary signal is detected, thereby determining that the target area is dangerous. In other words, boundary lines are set only at the boundary between the grass and the dangerous area. If a boundary signal is detected, indicating a danger, the mower cannot cross the boundary. If no boundary signal is detected, indicating no danger, the mower can cross the boundary.

[0100] In a possible implementation, obtaining the region type of each region includes:

[0101] Acquire a boundary signal, where the boundary signal is a magnetic field signal generated by a boundary line set at the boundary of the area to be driven;

[0102] Obtaining preset mapping information, where the preset mapping information represents a correspondence between the strength of the boundary signal and the area type;

[0103] The area type of each area is determined according to the strength of the boundary signal and preset mapping information.

[0104] It is understood that, depending on the area type of the target area adjacent to the area to be traveled, parameters of the boundary line between the area to be traveled and the target area can be set or controlled accordingly, so that when the area type of the target area is different, the strength of the magnetic field signal generated by the corresponding boundary line also varies accordingly. For example, when the area type of the target area is a safe area, the strength of the magnetic field signal generated by the corresponding boundary line can be controlled to be a first preset threshold, while when the area type of the target area is a dangerous area, the strength of the magnetic field signal generated by the corresponding boundary line can be controlled to be a second preset threshold, with the second preset threshold being greater than the first preset threshold. After obtaining the boundary signal generated by the boundary line set at the boundary of the area to be traveled, the mobile device can query the predetermined correspondence between the strength of the boundary signal and the area type or boundary type based on the strength of the boundary signal, thereby determining the area type of the target area or the boundary type of the target boundary. Specifically, for different area types corresponding to different areas adjacent to the area to be traveled, the strength of the magnetic field signal generated by the corresponding boundary line can be made different by controlling the input current to the boundary line between the area to be traveled and the adjacent area. For example, when the target area adjacent to the area to be driven is soil, the input current of the boundary line between the area to be driven and the soil area can be controlled to be a first current value, and when the target area adjacent to the area to be driven is stone, the input current of the boundary line between the area to be driven and the stone area can be controlled to be a second current value, and the second current value is greater than the first current value, so as to increase the strength of the magnetic field signal generated by the boundary line.

[0105] In one possible implementation, obtaining the area type of each area includes: obtaining the type of the boundary between the area to be driven and the target area, and determining the area type based on the boundary type. Specifically, map information is obtained, including the map, location information of each boundary in the map, and type information of each boundary in the map; first location information of each boundary is determined; and the boundary type is determined based on the first location information of each boundary and the map information.

[0106] Among them, the map information can be obtained by mapping based on satellite positioning sensors, inertial sensors and odometer sensors. Taking the mobile device as an automatic lawn mower as an example, refer to Figure 4. When the automatic lawn mower drives along the boundary of the grass area for the first time, the offset of each boundary (a, b, c, d, e) of the grass area and the charging station can be recorded according to the satellite positioning sensor, inertial sensor and odometer sensor, as shown in Figure 5; then, the user can set the type of each boundary (a, b, c, d, e) through the APP; finally, the offset of each boundary of the grass area and the charging station and the type of each boundary are stored in the map. Alternatively, the map information is obtained by mapping based on visual real-time positioning and map construction technology.

[0107] It is understood that users can pre-set the attributes of each boundary of the work area on the map. The boundary attributes include the location information and type information of the boundary, and mark and save the attribute information of each boundary on the map. For example, assuming the shape of the work area is rectangular, and the areas adjacent to the work area are stone, road, mud, and pool, the attribute information of the boundary corresponding to the work area can be set on the map as stone, road, mud, and pool, respectively, and the location information of each boundary can be marked on the map.

[0108] In one possible implementation, users can pre-divide the map into various areas, including the location of each area and the corresponding area type, and then mark and save the area division results, including the area type and corresponding location, on the map. For example, assuming that the map includes three adjacent areas: grass, mud, and a pool, the attribute information corresponding to each area can be set on the map, namely grass, mud, and a pool, and the location information of each area can be marked on the map. In this way, the mobile device can obtain the area type of the area to be driven and the target area based on the area type marked on the map during operation.

[0109] In this embodiment, the area type of each area in the captured image is obtained, and the mobile device can quickly identify various scenes such as roads, mud, fences, walls, etc., so as to select different working strategies for different scenes.

[0110] Step S102: When the area type of the target area is the first area type, controlling the mobile device to change the driving mode so that the coverage of the working unit includes a part of the target area.

[0111] In one embodiment of the present application, after obtaining the area type of each area in the captured image, the self-moving device can be controlled to travel and / or operate based on the area type of the target area. When the area type of the target area is a first area type, the self-moving device is controlled to change its travel mode so that the coverage of the operating unit includes a portion of the target area. When the area type of the target area is identified as a second area type, the self-moving device is controlled to travel and / or operate along the boundary of the area to be traveled. Changing the travel mode may include changing the travel path or changing the travel direction.

[0112] In order to enable the self-moving device to process the boundary of the working area while taking into account safety, the position of the working part can be adjusted according to the obtained area type so that the coverage of the working part meets the preset conditions, thereby improving the processing coverage rate at the boundary of the working area and enhancing the user experience. It can be understood that the preset conditions can be set according to actual needs. For example, taking the self-moving device as an automatic lawn mower, if it is determined according to the area type that the automatic lawn mower can cross the boundary or the working part can work at the boundary, the coverage of the working part is set to include a part of the target area, that is, it can cross the boundary, so that the automatic lawn mower can cut the grass at the boundary of the working area; if it is determined according to the area type that the automatic lawn mower cannot cross the boundary, the working part is set to travel within the working area, which can be along the boundary of the area to be traveled, that is, the working part cannot cross the boundary.

[0113] In an embodiment of the present application, if the target area is of a safe type or a first type of area in which the user expects the machine to travel, the coverage of the working unit can be controlled to include a portion of the target area. In this case, collisions, entanglements, jams, excessive tilting, falls, and other obstacles that hinder the movement of the self-moving device will not occur, and pedestrians or animals will not be harmed. If the target area is of an unsafe type or a second type of area in which the user does not expect the machine to travel, the machine can be controlled not to travel into the target area during travel to ensure the safety of the machine. For example, if the target area is a cobblestone road, the self-moving device can cross the boundary between the cobblestone road and the working area to safely travel and / or work on the cobblestone road. If the target area is a pool, the self-moving device cannot cross the boundary between the pool and the working area. It should be noted that the self-moving device can continue to operate during the adjustment of the working unit, or it can only operate after the adjustment of the working unit is completed. This is not specifically limited here.

[0114] In the present application, when the working mode is a preset working mode, when the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working part includes a part of the target area. The preset working mode is an edge working mode or a working mode preset by the user. The edge working mode is a mode for controlling the self-moving device to move and work along the boundary of the working area.

[0115] Specifically, the self-moving device includes at least a random operating mode and a side-by-side operating mode. In the random operating mode, the self-moving device randomly cuts through the work area or travels along a planned path. In the side-by-side operating mode, the self-moving device travels along the boundary of the work area. While traveling along the boundary, the device can execute the steps proposed in this application to obtain the region types of each region in the captured image and control the movement of the self-moving device based on the region types. When the conditions of the first region type are met, the working portion of the device can cover the portion of the target area that exceeds the work area, referred to as performing a side-by-side action. That is, in the side-by-side mode, while traveling along the boundary, the device can perform a side-by-side action, for example, traveling astride the boundary of the work area. Optionally, the self-moving device can also include a side-by-side operating mode. In this mode, the coverage area of ​​the working portion of the self-moving device includes a portion of the target area. In this case, the side-by-side operating mode can simply control the device to travel along the boundary within the work area. Furthermore, in the random operating mode, when the self-moving device is operating near the boundary of the work area, to enable the self-moving device to cover a larger working area, the side-by-side action can be triggered when the image meets the region type. Of course, in the present application, the cross-edge action can be triggered only when the cross-edge working mode is turned on, thereby improving the flexibility and convenience of controlling the mobile device.

[0116] As shown in Figure 9 , the self-moving device travels along the boundary of the work area within the work area. The self-moving device can move and operate within the work area enclosed by boundary 10, and can also move and operate along boundary 10. When the device moves along boundary 10, a certain distance d is maintained between the device and boundary 10, thereby preventing the device from crossing the boundary and being damaged. The device can determine the actual distance d between the device and boundary 10 based on the image.

[0117] FIG10 is a schematic diagram of a machine performing a side-crossing motion. A target area 70 is an area where the self-moving device can safely travel, such as a flagstone area, a sidewalk area, or a dirt area. When the self-moving device is located to the side of the target area 70, the target area exists in the image captured by the self-moving device. The to-be-marked area of ​​the target area can be marked as having a grass attribute, thereby changing the boundary between the grass attribute area and the target attribute area. For example, FIG7 shows a first image captured by the self-moving device, and FIG8 shows a second image after the area type adjacent to the to-be-moved area in the first image is marked as a grass type. It can be seen that the positions of the boundary 50 between the grass type area and the first area type in the first image and the boundary 60 between the grass type area and the first area type in the second image are different. The self-moving device moves along the edge according to the boundary 60 between the grass type area and the area containing the first area type in the second image, thereby moving across the boundary 10 of the actual space, achieving cutting to the edge and completely cutting the grass between the working area and the flagstone area.

[0118] In one possible implementation, the self-mobile device can provide multiple operating modes for the user to select, or automatically select the desired operating mode during operation. Based on the operating mode selected by the user, when the operating mode is edge-based, the machine can execute the steps of obtaining the region type of each region in the captured image and controlling the movement of the self-mobile device based on the region type. Alternatively, the machine can execute the corresponding process according to the execution flow planned internally. For example, the machine can first execute the random operating mode, and after completing the corresponding operation in the random operating mode, it can switch to the edge-based working mode.

[0119] In one embodiment of the present application, when the area type of the target area is identified as the first area type, the self-moving device is controlled so that the coverage range of its working part includes a portion of the target area. In one possible implementation, the first area type includes one or more of roads, manhole covers, stone roads, hard roads, fallen leaves, and mud. Among them, roads include but are not limited to cement floors, asphalt floors, roads, sidewalks, etc. The first area type is not limited to this, as long as the self-moving device can safely drive and / or work on it. The first area type can also be set according to received user instructions, so that the machine can perform edge crossing in certain scenarios according to user needs. The self-moving device can identify multiple types of objects or objects in the image based on the image recognition model, for example, it can simultaneously identify grass, ditches, puddles, fences, stone slabs, fallen leaves, walls, and other non-grass objects; the user can select one or more types as the first area type, for example, the user can select stone slabs and fallen leaves as the first area type.

[0120] In one embodiment of the present application, when the area type of the target area is the second area type, the self-moving device is controlled to travel and / or work along the boundary of the area to be traveled. The second area type may include one or more of a wall, a pit, a pool, a fence, a flower bed, soil with a soil hardness less than a hardness threshold, an irrigation facility area, and a movable life area. Furthermore, when the second area type is met, if the height difference between the area where the second area type is located and the area to be traveled is equal to or greater than the height threshold (for example, 10 cm), the machine is controlled to travel along the boundary of the area to be traveled to ensure the safety of the machine. When it is identified that the area type of the target area is the second area type or an area with a higher height difference, it is necessary to control the machine to remain in the working area to prevent rollover or crashing, or unsafe phenomena such as collision of the working part.

[0121] For example: As shown in FIG6 , in one embodiment, the image captured by the image acquisition device can be detected based on the image recognition model. When an obstacle area 30 (i.e., the area where the second area type in the present application is located) is detected, the self-moving device is controlled to move away from the boundary between the obstacle area 30 (such as a pool) and the working area, and the self-moving device is controlled to maintain a safety distance f from the boundary. The self-moving device moves and works while maintaining the safety distance f to avoid damage to the machine. The value of the safety distance f can be fixed or adjustable. The value of the safety distance f can be automatically adjusted by the machine or set or adjusted by the user.

[0122] In one embodiment of the present application, as shown in Figures 7 and 8, controlling the self-moving device to travel and / or work based on the area type of the target area so that when the area type of the target area is the first area type, the coverage of the working unit includes a part of the target area, which may include: marking the area type of the part of the target area adjacent to the area to be traveled as a grassland type; and controlling the self-moving device to travel and / or work along the boundary between the area determined by the grassland type and the area determined by the first area type.

[0123] Specifically, as shown in FIG7 , the area to be traveled is of a grass type. When the area type of the target area is of the first area type, the area to be traveled can be expanded as shown in FIG8 , and the intersection area between the grass area obtained after expansion and the target area is marked as a grass attribute (i.e., grass type) to obtain a second image. Alternatively, an area to be marked can be determined in the target area, and the portion of the target area close to the grass area is used as the area to be marked. The area to be marked is then marked with grass attributes to obtain a marked grass area to obtain a second image. In this way, the self-moving device can be controlled to move along the boundary between the grass area and the target area in the second image. In an embodiment of the present application, since the boundary position between the grass area and the target area is adjusted, the self-moving device can move along the adjusted boundary, and the boundary is substantially consistent with the physical boundary in the real space. Therefore, the self-moving device can move across the physical boundary in the real space and cut during the movement, thereby cutting the grass at the boundary cleanly, achieving the effect of cutting to the edge, and improving the cutting coverage rate and user satisfaction.

[0124] Furthermore, obtaining the area type of each area in the captured image may further include: performing depth estimation processing on the image to determine depth information corresponding to the target area, where the depth information is used to represent the distance between each position in the target area and the image capture device. Accordingly, controlling the movement and / or operation of the self-moving device based on the area type of the target area may include: controlling the movement and / or operation of the self-moving device based on the area type of the target area and the depth information, such that when the area type of the target area is a first area type and the height difference between the target area and the area to be traveled is less than a height threshold, the coverage range of the working unit includes a portion of the target area. That is, in addition to meeting the area type requirement, the target area also needs to meet the height requirement. When the height difference between the target area meeting the first area type and the area to be traveled is less than a height threshold, such as the height threshold can be set to 3 cm, 5 cm, etc. For example, if the target area is a manhole cover with a height difference of 3 cm from the working area, the self-moving device can be controlled to perform a side-crossing action.

[0125] In one embodiment of the present application, as shown in Figures 13 and 14 , when a self-moving device enters edge-along operation mode and identifies a stone slab 70 crossing an edge, if a collision with a stone 81 is detected, the self-moving device can be controlled to change its direction and move toward the interior of the work area, as shown in Figure 14 . Moving toward the interior of the work area can include moving along the boundary of the work area or controlling the projection of the working part within the work area. For example, if a lawn mower detects camera displacement while moving across an edge, it can be controlled to move within the work area.

[0126] Furthermore, when a collision is detected, the time or distance the machine travels into the work area is recorded. If the distance exceeds 5 meters, the machine can be controlled to restart to identify whether the first area type exists. If so, the machine will continue to straddle the boundary. If no stone slab is detected during the edge-travel process, the machine will be controlled to move into the work area. If the distance traveled exceeds 5 meters, the machine can be controlled to restart straddling the boundary.

[0127] In an application scenario of the present application, during the edge-crossing process, when the machine walks to the end of the long strip of cobblestone road, only part of the cobblestone road remains in the picture. Since the machine uses single-frame recognition, the block-shaped cobblestone road in the image will be considered as stepping stones. Stepping stones refer to stone slabs that are relatively small and high and cannot be passed by machines. Therefore, due to recognition errors, the machine will not be able to continue along the edge. To address this problem, it is possible to first determine whether the machine is already in a stable edge-crossing state. If the machine is in a stable edge-crossing state, the stepping stones are used as cobblestone roads and the edge-crossing action is performed; if it is not in a stable edge-crossing state, the processing logic of avoiding stepping stones, that is, controlling the machine to travel into the working area, is executed. Among them, if the boundary of the cobblestone road is seen for a continuous period of time (such as within 2S), the stable edge-crossing state is entered; if the boundary of the cobblestone road is not seen for a continuous period of time (such as within 5S), the stable edge-crossing state is exited.

[0128] In one embodiment, the first image captured by the image acquisition device may contain one or more of a grass area, a target area of ​​the first area type, and a target area of ​​the second area type (hereinafter also referred to as an obstacle area).

[0129] In one embodiment of the present application, when a target area in a first image includes multiple area types, the autonomous vehicle can be controlled to travel and / or operate based on the area type of the target area, such that the autonomous vehicle avoids areas of a second area type during travel. This prevents the autonomous vehicle from partially traveling into areas of the second area type when traveling along the boundary between an area defined by a grass type and an area defined by the first area type, potentially causing a safety issue. The target area includes a first area type and a second area type, and both the area of ​​the first area type and the area of ​​the second area type are adjacent to the area to be traveled.

[0130] In one embodiment of the present application, the self-moving device is controlled to travel and / or operate based on the target area's area type, such that the travel distance of the operating unit is less than the length of the boundary between the area of ​​the first area type and the area to be traveled. The travel distance is the distance the self-moving device would travel if the coverage of the operating unit included a portion of the target area. That is, the travel distance of the self-moving device in response to identifying the first area type is less than the length of the physical boundary actually adjacent to the first area type. If multiple target area types exist, the machine is controlled to only perform some of the cross-border actions to prevent safety issues.

[0131] In one embodiment of the present application, while a self-propelled device is traveling along a first path, images are captured and the area type of each area in the image is determined. The self-propelled device is then controlled to continue traveling along the first path for a period of time. After the period of travel, the path is changed so that the working unit's coverage area includes a portion of the target area. Alternatively, based on the length of the mower, the mower can be controlled to travel to a position a distance equal to the length of the mower from the pool, whereupon the controller performs the side-crossing action.

[0132] For example, the scenario shown in Figures 11 and 12 illustrates this. While driving in its current direction, a lawn mower obtains the area types of various regions in the image, including grass 90, cobblestone path 70, and pond 80. The cobblestone path is adjacent to the pond, and the entire cobblestone path and pond are adjacent to the work area. The lawn mower is controlled to activate border-crossing mode. The camera captures the image, and semantically identifies the grass, cobblestone path, and pond in the image. The grass in the image is expanded, and the portion of the expanded image that overlaps with the cobblestone path is converted to grass. The cobblestone path and pond in the non-overlapping portion are marked as non-grass. As shown in the figure, if the mower directly reaches the cobblestone position PP adjacent to the pond, its rear end may enter the pond. Therefore, the mower can be controlled to continue driving in its current direction for a period of time before changing direction to follow the boundary between grass and non-grass in the image. By delaying the response to the image recognition result, the mower will not fall into the pond when it approaches the grass area in the image, ensuring its safety.

[0133] In the present application, the area type of each area in the captured image is obtained. When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage of the working part includes a part of the target area. The self-moving device can process the boundaries of the working area while taking safety into consideration, thereby improving the processing coverage rate of the boundaries of the working area and enhancing the user experience. Furthermore, the self-moving device can quickly identify the area type of each area in the image, that is, it can quickly identify multiple scenes, so that different working strategies can be selected for different scenes.

[0134] In another embodiment of the present application, the self-mobile device collects images during driving and establishes a local map of its nearby area based on the images. When a scenario is identified in which the area where the first area type and the area where the second area type are located are adjacent to the area to be driven, the driving path of the self-mobile device can be planned according to the local map to control the avoidance of the area where the second area type is located after driving according to the adjusted image.

[0135] In one embodiment, after obtaining an image captured by an image acquisition device (hereinafter also referred to as a first image), marking the area type of a portion of the target area adjacent to the area to be driven as a grass type to obtain a second image includes:

[0136] Traversing all pixels in the first image to determine pixels in the area to be marked;

[0137] The pixels in the area to be marked are marked as pixels corresponding to grass, thereby obtaining a second image.

[0138] For example, the color of the pixels in the area to be marked may be modified to green, or the pattern shape of the area to be marked may be modified to the shape of grass.

[0139] The area to be marked may be a partial area of ​​the target area with a preset width adjacent to the grass area.

[0140] In one embodiment, after obtaining an image captured by the image acquisition device (hereinafter also referred to as a first image), marking the area type of a portion of the target area adjacent to the area to be driven as a grassland type includes:

[0141] Performing rasterization processing on the first image to generate a raster image, the raster image including a grassland raster corresponding to the grassland area and a target raster corresponding to the target area, the attributes of the grassland raster being grassland attributes, and the attributes of the target raster being the first area type or the second area type;

[0142] Perform expansion processing on the grassland raster, and mark the attributes of the expanded raster as grassland attributes;

[0143] Get the overlapping part of the target grid and the expanded grid;

[0144] If the attribute of the overlapping grid in the target grid is the first area type, the attribute of the overlapping grid is modified to the grassland attribute;

[0145] If the attribute of the overlapping grid in the target grid is the second region type, the attribute of the overlapping grid is modified to a non-grassland attribute to obtain a second image.

[0146] Alternatively, after generating the grid image, the grid to be marked may be determined according to the position of the grassland grid and the position of the target grid; and the attributes of the grid to be marked are marked as grassland attributes to obtain the second image.

[0147] The standard for rasterizing an image is: a grid contains 25 pixels, each pixel is marked with a corresponding attribute, and each grid attribute is based on the attribute of the larger number of pixels. The first image can be rasterized based on the intrinsic parameters, extrinsic parameters, distortion parameters, and image pixels of the camera to obtain a raster image corresponding to the first image. During the rasterization process, the grassland area can be converted into a grassland grid, and the attributes of the grassland grid are grassland attributes; the target area can be converted into a target grid, and the attributes of the target grid are target attributes, that is, the first region type or the second region type. Non-grassland attributes include the first region type or the second region type. In this embodiment, the target attribute of the grid to be marked can be modified to the grassland attribute to obtain the second image.

[0148] In one embodiment, the grid adjacent to the grid to be marked includes a grass grid. The grid to be marked may be a grid adjacent to the grass grid. For example, if any grid among the adjacent grids around the grid to be marked is a grass grid, the grid to be marked may be regarded as a grass grid.

[0149] In one embodiment, when the mobile device moves along the boundary of the working area in a counterclockwise direction, the left grid of the grid to be marked is a grass grid.

[0150] When the mobile device moves counterclockwise along the boundary of the working area, the boundary is on the right side of the mobile device and the grassland is on the left side of the mobile device. Therefore, it can be determined whether the left grid of the target grid is a grassland grid. If it is a grassland grid, the target grid can be used as the grid to be marked.

[0151] When the mobile device moves along the boundary of the working area in a clockwise direction, it is determined whether the grid to the right of the target grid is a grassland grid. If it is a grassland grid, the target grid can be used as a grid to be marked.

[0152] In one embodiment, the lower grid of the grid to be marked is a grass grid.

[0153] For example, a stone slab may be located directly in front of the machine, and the camera may not capture the side of the stone slab parallel to the machine's forward direction, but only the side of the stone slab perpendicular to the machine's forward direction. In this case, since the grid to the left of all target grids is the target area, it can be determined whether the grid below the target grid is a grass grid. If it is a grass grid, the target grid can be used as the grid to be marked. Alternatively, in another embodiment, if the to-be-traveled area does not exist in the image, it indicates that the machine is at a right angle to the boundary. If the machine is traveling counterclockwise along the boundary, the machine is controlled to turn left or an alarm is issued.

[0154] Since the position and shape of the machine's boundaries are diverse, the position and shape of the boundaries in the image captured by the machine are also diverse. You can set specific conditions according to actual needs and select the target grids that meet the conditions as the grids to be marked.

[0155] In one embodiment, determining the grid to be marked according to the positions of the grassland grid and the target grid includes:

[0156] The grid to be marked is determined based on the position of the grassland grid, the position of the target grid and the cross-edge distance conditions.

[0157] For example, the span distance condition may represent the width of a portion of the machine body that exceeds an actual boundary, or the width of a portion of the machine working part that exceeds a boundary.

[0158] Based on the positions of the grassland grid and the target grid, the target grid adjacent to the grassland grid can be determined and used as the first column of grids to be marked. Furthermore, based on the cross-edge distance condition, the target grids located to the sides of the first column of grids to be marked can be selected as the other columns of grids to be marked. The value of X can be determined based on the cross-edge distance condition.

[0159] For example, if the width of the portion of the machine body that exceeds the actual boundary is required to be 10 cm, and the width of a grid is 5 cm, two columns of target grids can be used as grids to be marked, and X is 1.

[0160] For example, if the control logic of the machine requires that the body of the machine maintain a certain distance from the corresponding position of the boundary in the image, such as 10 cm, and the cross-edge distance condition requires that the width of the part of the machine body that exceeds the actual boundary is 10 cm, and the width of a grid is 5 cm, then the four columns of target grids that meet the conditions can be used as the grids to be marked, and X is 3.

[0161] In one embodiment, after converting the first image to obtain a first raster image, all the grids in the first raster image can be traversed to determine the grids to be marked in the first raster image, mark them as grass attributes, and obtain a second raster image. The grids to be marked in the second raster image are then determined and marked as grass attributes, obtaining a third raster image. The grids to be marked in the third raster image are then marked as grass attributes, obtaining a fourth raster image. The grids to be marked in the fourth raster image are then marked as grass attributes, obtaining a fifth raster image, i.e., the second image. In this manner, all the grids in the image are traversed four times, achieving the goal of using four columns of target grids as grids to be marked.

[0162] In one embodiment, during the process of traversing the grids, the grass grid originally having grass attributes will not be modified, and the obstacle grid originally having obstacle attributes will not be modified.

[0163] In one embodiment, controlling the self-moving device to travel and / or operate according to the second image includes:

[0164] Based on the second image, the self-moving device is controlled to travel and / or work in the space corresponding to the grass attribute area, avoid the space corresponding to the non-grass attribute area, and control the working part coverage range of the self-moving device to meet preset conditions; the preset conditions include that the working part coverage range includes a part of the target area, or the working part coverage range intersects with the boundary of the working area, or the distance between the working part coverage range and the boundary of the working area is less than a first threshold.

[0165] For example, in the process of moving along the boundary in a counterclockwise direction, when the self-mobile device detects the target area based on the image, the self-mobile device can be controlled to turn right and move according to the processed second image until the self-mobile device drives and / or works in the space corresponding to the grass attribute area of ​​the second image, and drives along the position corresponding to the boundary between the grass attribute area and the target attribute area, thereby realizing cross-edge cutting.

[0166] As shown in Figure 10, in real space, the autonomous machine is driving and cutting across the edge; however, its internal control logic still believes it is driving and cutting along the boundary. This is because the grass attribute area and the target attribute area in the second image are re-labeled based on the first image. This approach is also more compatible with other control logic of the machine.

[0167] In one embodiment, controlling the self-moving device to travel and / or operate according to the second image includes:

[0168] Based on the second image, a map of the working area is generated, the map including a drivable area and a non-drivable area, the drivable area corresponds to the grass attribute area of ​​the second image, and the non-drivable area corresponds to the non-grass attribute area of ​​the second image; based on the map, the self-moving device is controlled to drive and / or work so that the self-moving device drives and / or works in the space corresponding to the drivable area and avoids the space corresponding to the non-drivable area.

[0169] A map of a local area within the work area is generated based on the second image. This map can reflect the environment surrounding the mobile device, including the grass and non-grass areas surrounding the mobile device. The drivable area corresponds to the grass-attributed area in the second image, including the actual grass area and the target area whose attributes have been modified to grass. The non-drivable area corresponds to the non-grass-attributed area in the second image, including the actual obstacle area, the target area whose attributes have not been modified, and the obstacle area.

[0170] The self-moving device can plan its path according to the map, thereby moving along the boundary between the drivable area and the non-drivable area, realizing cross-edge cutting in the actual space.

[0171] In one embodiment, the height difference between the target area and the grass area is smaller than a second threshold.

[0172] In this embodiment, the target area is a safe area, so the height difference between the two cannot be too large.

[0173] As shown in FIG9 , in one embodiment, the method further includes:

[0174] When the region type of the target region in the first image is the second region type, obtaining edge distance information;

[0175] According to the edge distance information, the distance d between the self-moving device and the boundary 10 of the working area is controlled.

[0176] If only the second area type is identified in the images captured by the self-driving device while moving along the edge, it indicates that the area outside the boundary between the travel area and the target area may be dangerous, and therefore movement across the boundary is prohibited. To ensure safety, a certain distance d can be maintained between the machine and the boundary 10. This distance can be fixed or adjustable.

[0177] The actual distance d may be adjusted according to the acquired edge distance information so that the actual distance d is equal to the distance value represented by the edge distance information.

[0178] In one embodiment, obtaining the edge distance information includes: obtaining user instruction information; and determining the edge distance information based on the user instruction information. The edge distance information can be determined based on the user instruction. For example, if the user selects 10 cm, the edge distance information is 10 cm. The mobile device can adjust its movement direction to adjust the distance d between itself and the boundary 10 to 10 cm.

[0179] In one embodiment, the area type of the portion of the target area adjacent to the area to be driven is marked as a grass type to obtain a second image, including: determining the area to be marked; and when no collision is detected, marking the attributes of the area to be marked in the first image as grass attributes to obtain a second image.

[0180] In one embodiment, the attributes of the area to be marked in the first image are marked as grass attributes to obtain a second image, including: determining the area to be marked; in the case of detecting a collision, determining a first area and a second area of ​​the area to be marked, the distance between the first area and the grass area being less than the distance between the second area and the grass area; marking the attributes of the first area as grass attributes to obtain a second image.

[0181] Exemplarily, the machine is provided with collision detection sensors.

[0182] For example, when the machine does not have the cross-edge cutting function (i.e., the cross-edge action described above) enabled, the machine's control logic requires that the machine body maintain a certain distance, such as 15 cm, from the position corresponding to the boundary in the image. When the cross-edge cutting function is enabled and the machine does not detect a collision, the machine's control logic requires that the machine body maintain a certain distance, such as 10 cm, from the position corresponding to the boundary in the image. The cross-edge distance condition requires that the width of the portion of the machine body that exceeds the actual boundary be 10 cm, and the width of a grid is 5 cm. In this case, the four target grids that meet the conditions can be used as grids to be marked.

[0183] When the collision sensor of the machine detects a collision, the machine can be controlled to move away from the collision object, increase the distance between the machine and the collision object, and move. For example, when the cross-edge cutting function of the machine is turned on and the collision sensor of the machine detects a collision, the control logic of the machine is that the body and the corresponding position of the boundary in the image need to maintain a certain distance, such as 10 cm, and the cross-edge distance condition can be changed to require that the width of the part of the machine body that exceeds the actual boundary is 0 cm, and then the two columns of target grids that meet the conditions can be used as grids to be marked. That is to say, when cross-edge cutting is turned on and a collision is detected, the number of columns of grids to be marked can be reduced so that the number of columns is less than the number of columns of grids to be marked when cross-edge cutting is turned on and no collision is detected.

[0184] In one embodiment, the area type of the portion of the target area adjacent to the area to be driven is marked as a grass type, including: determining the area to be marked; determining a first area and a second area of ​​the area to be marked, wherein the distance between the first area and the grass area is smaller than the distance between the second area and the grass area; marking the attributes of the first area and the second area as grass attributes; and in the event of a collision being detected, marking the attributes of the second area as target attributes to obtain a second image.

[0185] When cross-edge cutting is enabled and a collision is detected, all grids to be marked may be marked as grass grids, and then a portion of the grids to be marked may be marked back to the target attribute, thereby obtaining a second image.

[0186] In one embodiment, when the self-moving device does not detect a collision, the step of marking the attributes of the area to be marked in the first image as grass attributes to obtain a second image is executed; the control method also includes: when the self-moving device detects a collision, marking the attributes of the area to be selected in the first image as target attributes to obtain a third image, where the area to be selected is an area in the grass area close to the target area; based on the third image, the self-moving device is controlled to travel and / or work.

[0187] After the cross-edge cutting function is turned on, if no collision is detected, the attributes of the area to be marked can be directly marked as grass attributes to obtain a second image.

[0188] When the cross-edge cutting function is enabled and a collision is detected, the grid cells in the grass area near the target area are marked as target attributes, generating a third image. The autonomous vehicle can then be controlled based on this third image to move the boundary inward toward the grass, moving the autonomous vehicle away from the object and avoiding a collision. The location of the retracted boundary can be directly analyzed based on the third image to control the vehicle. Alternatively, a grid map can be generated based on the third image, and the vehicle can be controlled based on the boundary information in the map.

[0189] In one embodiment, for a first image acquired within a preset time period after a collision is detected, attributes of a to-be-selected region in the first image are marked as target attributes to obtain a third image.

[0190] For example, within 7 seconds after the collision, the step of marking the attributes of the area to be selected in the first image as the target attributes is performed to obtain the third image. After 7 seconds after the collision, the step of marking the attributes of the area to be marked in the first image as the grass attributes is performed to obtain the second image.

[0191] For example, when the cross-edge cutting function is not enabled on the machine, the control logic of the machine is that the machine body needs to maintain a certain distance from the position corresponding to the boundary in the image, such as 15 cm.

[0192] When the machine turns on the cross-edge cutting function and does not detect a collision, the machine's control logic requires that the machine body maintain a certain distance from the corresponding position of the boundary in the image, such as 10 cm. The cross-edge distance condition requires that the width of the part of the machine body that exceeds the actual boundary is 10 cm, and the width of a grid is 5 cm. In this case, the four columns of target grids that meet the conditions can be used as grids to be marked.

[0193] When the machine's cross-edge cutting function is turned on and the machine's collision sensor detects a collision, the machine's control logic requires that the machine body maintain a certain distance from the corresponding position of the boundary in the image, such as 10 cm. The cross-edge distance condition is changed to require that the machine body retract into the actual boundary by a width of 10 cm. In this way, the two columns of target grids in the grass area can be used as candidate grids.

[0194] That is to say, when the cross-edge cutting function is turned on and a collision is detected when crossing the edge, the self-moving device can be controlled to move inward along the edge within a period of time after the collision, that is, the self-moving device can be controlled to move along the edge inside the grass, and the distance between the self-moving device and the boundary can be appropriately increased to avoid collision.

[0195] It should be noted that the different embodiments provided in the present disclosure can be combined with each other, but the manner of combination is not specifically limited.

[0196] An embodiment of the present disclosure also provides a self-moving device, comprising: a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the control method of the self-moving device described in any one of the embodiments of the present disclosure is implemented.

[0197] Based on the same inventive concept as the above embodiment, this embodiment further provides a computer storage medium, wherein the computer storage medium stores a computer program. The computer storage medium may be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); or various devices including one or any combination of the above memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc. When the computer program stored in the computer storage medium is executed by a processor, the control method of the self-moving device applied to the above device is implemented. For the specific steps implemented when the computer program is executed by the processor, please refer to the description of the embodiment shown in Figure 1, which will not be repeated here.

[0198] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] As used herein, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion of elements other than the listed elements and may also include additional elements not specifically listed.

[0200] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for controlling a self-moving device, the self-moving device comprising a working unit configured to perform a work operation, the self-moving device also comprising: include: An image acquisition device, wherein the image acquisition device is configured to acquire an image in front of the self-moving device, and the self-moving device is configured to travel and / or work in a working area, wherein the method comprises: Acquire the area type of each area in the acquired image, wherein each area at least includes: an area to be driven and a target area, wherein the area to be driven is a part of the working area, and the target area is an area adjacent to the area to be driven; When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working unit includes a part of the target area.

2. The method according to claim 1, It is characterized in that The method further comprises: The collected image is input into the trained image recognition model to perform semantic segmentation on the image to obtain the region type of each region.

3. The method according to any one of claims 1 to 2, It is characterized in that The method further comprises: When the working mode is a preset working mode, the step of controlling the self-moving device to change the driving mode so that the coverage range of the working unit includes a part of the target area is executed when the area type of the target area is the first area type. The preset working mode is an edge working mode or a working mode preset by the user. The edge working mode is a mode for controlling the self-moving device to move and work along the boundary of the working area.

4. The method according to any one of claims 1 to 3, It is characterized in that The first area type includes one or more of roads, manhole covers, stone roads, hard roads, fallen leaves and mud, or the first area type is obtained according to user instructions.

5. The control method according to any one of claims 1 to 4, It is characterized in that The method further comprises: When a contact collision is detected during the process in which the working part covers a portion of the target area, the self-moving device is controlled to change a driving path so as to drive toward the inside of the working area.

6. The method according to any one of claims 1 to 5, It is characterized in that Obtaining the region type of each region in the acquired image also includes: Performing depth estimation processing on the image to determine depth information corresponding to the target area, the depth information being used to represent the distance between each position of the target area and the image acquisition device; Correspondingly, when the area type of the target area is the first area type, controlling the self-moving device to change the driving mode includes: The self-moving device is controlled to change the driving mode according to the area type of the target area and the depth information, so that when the area type of the target area is the first area type and the height difference between the target area and the area to be driven is less than a height threshold, the coverage range of the working unit includes a part of the target area.

7. The method according to any one of claims 1 to 6, It is characterized in that The image is semantically segmented to obtain the region type of each region, the region type of the to-be-traveled region is a grassland type, and the region type of the target region is a first region type; accordingly, Controlling the coverage of the working portion to include a portion of the target area comprises: Marking the area type of the portion of the target area adjacent to the area to be driven as a grassland type; The self-moving device is controlled to travel and / or operate along a boundary between an area defined by the grass type and an area defined by the first area type.

8. The method according to claim 1, It is characterized in that When the area type of the target area is the second area type, the self-moving device is controlled to travel and / or work in the working area.

9. The method according to claim 8, wherein the method comprises: It is characterized in that The second area type includes one or more of a wall, a pit, a pool, a fence, a flower bed, soil with a hardness less than a hardness threshold, an irrigation facility area, and a movable life form area.

10. The control method according to claim 8 or 9, It is characterized in that Controlling the self-moving device to travel and / or work in the working area includes: Get the distance information along the edge; The self-moving device is controlled to travel and / or work in the working area according to the edge distance information.

11. The control method according to claim 10, It is characterized in that Get edge distance information, including: Get user command information; The edge distance information is determined according to the user instruction information.

12. The control method according to any one of claims 1 to 11, It is characterized in that The method further comprises: When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the self-moving device avoids the area where the second area type is located, wherein the area type of the target area includes the first area type and the second area type, and the area where the first area type is located and the area where the second area type is located are both adjacent to the area to be driven.

13. The control method according to any one of claims 1 to 12, It is characterized in that The method further comprises: When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the driving distance of the working unit is less than the boundary length between the area where the first area type is located and the area to be driven, wherein the driving distance is the distance traveled by the self-moving device when the coverage range of the working unit includes a part of the target area.

14. The control method according to any one of claims 1 to 13, It is characterized in that Before obtaining the area type of each area in the collected image, the self-moving device travels along a first path; When the area type of the target area is the first area type, controlling the self-moving device to change the driving mode includes: The self-moving device is controlled to continue traveling along the first path for a period of time, and after traveling for a period of time, the traveling path is changed so that the coverage range of the working unit includes a part of the target area.

15. A self-moving device, the self-moving device include: A working unit configured to perform a work operation; An image acquisition device, wherein the image acquisition device is configured to acquire an image in front of the mobile device; a controller, connected to the working part and the image acquisition device by signal, the controller being configured to control the self-moving device to travel and / or work in a working area, The invention is characterized by comprising: The controller acquires the area type of each area in the acquired image, wherein each area at least includes: an area to be driven and a target area, wherein the area to be driven is a part of the working area, and the target area is an area adjacent to the area to be driven; When the area type of the target area is the first area type, the self-moving device is controlled to change the driving mode so that the coverage range of the working unit includes a part of the target area.

16. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the control method of a self-mobile device according to any one of claims 1 to 14.

Images