Missed operation area detection method of mobile equipment, robot and storage medium

Abstract

The invention provides a missed operation area detection method of mobile equipment, a robot and a computer readable storage medium. The method comprises the following steps: acquiring an image corresponding to an operation area where the mobile equipment performs mobile operation; carrying out preliminary judgment on a missed operation area based on the image, and determining a candidate missed operation area; obtaining the area or perimeter of the candidate leakage operation area, and if the candidate leakage operation area meets a preset condition, determining the candidate leakage operation area as a first target leakage operation area; and in a mobile operation process, performing supplementary operation processing based on the first target operation leakage area. According to the embodiment of the invention, the reliability of mobile operation of the mobile equipment is improved.

Description

Technical Field

[0001] This application relates to the field of navigation technology, and in particular to a method for detecting missed work areas of a mobile device, a robot, and a storage medium. Background Technology

[0002] With the development of technology, mobile operations using robots have become a trend. For example, lawn mowing robots autonomously move along planned paths to mow the lawn. However, during operation, issues such as robot positioning drift, wheel slippage, and path tracking errors can lead to missed mowing spots. Summary of the Invention

[0003] This application provides a method for detecting missed work areas of a mobile device, a robot, and a storage medium, aiming to improve the reliability of mobile devices performing mobile operations.

[0004] To achieve the above objectives, this application provides a method for detecting missed work areas in a mobile device, comprising: Acquire images of the work area corresponding to the mobile operation performed by the mobile device; Based on the image, a preliminary determination of the missed work area is made to identify candidate missed work areas; Obtain the area or perimeter of the candidate missed work area. If the candidate missed work area meets the preset conditions, then determine the candidate missed work area as the first target missed work area. During the mobile operation, supplementary operations are performed based on the first target missed operation area.

[0005] Furthermore, to achieve the above objectives, this application also provides a robot, which includes a walking device, an image acquisition device, and a control device; wherein, the walking device is used to move the robot; the image acquisition device is used to acquire images corresponding to the work area where the robot performs its movement operations; the control device includes a processor and a memory, the memory storing a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements the steps of the above-described method for detecting missed work areas of a mobile device.

[0006] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the above-described method for detecting missing work areas in a removable device.

[0007] The missed work area detection method, robot, and storage medium for mobile devices provided in this application embodiment acquire images corresponding to the work area of ​​the mobile device performing mobile operations, make a preliminary judgment on the missed work areas based on the images, determine candidate missed work areas, and obtain the area or perimeter of the candidate missed work areas. If the candidate missed work areas meet preset conditions, the candidate missed work areas are determined as the first target missed work areas. During the mobile operation, supplementary work is performed based on the first target missed work areas, thereby solving the missed work problem and improving the reliability of mobile operations performed by mobile devices.

[0008] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating a method for detecting missed work areas in a mobile device according to an embodiment of this application. Figure 2 This is a schematic block diagram of a mobile device provided in an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of a lawnmower robot performing lawnmowing operations, as provided in an embodiment of this application. Figure 4 This is a schematic diagram of a process for determining candidate missed job regions provided in an embodiment of this application; Figure 5 This is a schematic diagram of a process for obtaining the area or perimeter of the candidate missed work area provided in an embodiment of this application; Figure 6 This is a schematic diagram of a process for performing supplementary work based on the first target missed work area, provided by an embodiment of this application; Figure 7 This is a schematic flowchart illustrating a method for obtaining the location information of the first target missed work area, as provided in an embodiment of this application. Figure 8 This is a schematic diagram of the structure of a robot provided in an embodiment of this application; Figure 9 This is a schematic block diagram of a robot control device provided in an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0013] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0014] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] The embodiments of this application provide a method for detecting missed work areas of a mobile device, a robot, and a storage medium to improve the reliability of mobile devices performing mobile operations.

[0016] Please see Figure 1 , Figure 1 This is a flowchart illustrating a method for detecting missed work areas in a mobile device according to an embodiment of this application. This method can be applied to robots or robot control devices (such as remote controls, mobile phones, etc.), and this application does not limit the application scenarios of this method.

[0017] like Figure 1 As shown, the method for detecting missing work areas in the mobile device specifically includes steps S101 to S104.

[0018] S101. Obtain the image corresponding to the work area of ​​the mobile device for mobile operations.

[0019] Mobile devices include, but are not limited to, lawnmower robots, autonomous vehicles, indoor mobile robots, and patrol robots. It is understandable that different mobile devices correspond to different work areas; for example, a lawnmower robot might work on an outdoor lawn, while an indoor mobile robot might work on an indoor floor.

[0020] The image corresponding to the working area of ​​the mobile device can be a raw image / video frame captured by a camera, webcam, etc., or it can be an image obtained after preprocessing such as distortion correction of the raw image / video frame. No specific limitation is made in this application.

[0021] For example, such as Figure 2 As shown, the mobile device 1000 includes an image acquisition module 100 and a data processing module 200. The image acquisition module 100 includes, but is not limited to, a camera or webcam, and can continuously acquire images of the work area at a corresponding frequency (e.g., 5-10 frames per second). For example, the image acquisition module 100 may be a high-definition wide-angle camera with a certain downward viewing angle installed on the mobile device 1000, which can simultaneously capture images of both near and far areas in front of the mobile device 1000. Furthermore, the mobile device 1000 may also be equipped with a supplementary lighting device (e.g., an LED supplementary light) to address insufficient light in early morning, dusk, or shaded areas, thereby ensuring the quality of the acquired images.

[0022] The data processing module 200 includes, but is not limited to, a central processing unit. The data processing module 200 can be an embedded computing platform (such as an NVIDIA Jetson series or similar AI chip). The data processing module 200 is responsible for receiving image data acquired by the image acquisition module 100 and preprocessing the received images to eliminate environmental interference. Preprocessing includes, but is not limited to, distortion correction, color correction, and illumination equalization.

[0023] For example, taking a mobile device like a lawnmower robot as an example, the robot is equipped with a high-definition wide-angle camera at its front. This camera can simultaneously capture the condition of the lawn both near and far in front of the robot, and can better perceive differences in lawn height through perspective. When the lawnmower robot is mowing, such as... Figure 3 As shown, the lawn images are first continuously captured by the camera on the lawnmower at a corresponding frequency (e.g., 7 frames / second), and the captured lawn images are preprocessed by distortion correction, color correction, and illumination equalization to eliminate environmental interference and obtain high-quality lawn images.

[0024] S102. Based on the image, a preliminary determination of the missed work area is made to identify candidate missed work areas.

[0025] During mobile operations, missed tasks may occur. By performing feature extraction and region segmentation on the image of the work area, the missed task areas are initially identified. These initially identified missed task areas are then used as candidate missed task areas, which are areas that need to be further determined to be missed task areas.

[0026] In some embodiments, such as Figure 4 As shown, step S102 may include sub-step S1021 and sub-step S1022.

[0027] S1021. Input the image into the trained semantic segmentation model and output the segmentation map corresponding to the image; the segmentation map includes a first type of region and a second type of region, the first type of region represents the unfinished operation area in the mobile device's operation range, and the second type of region represents the completed operation area. S1022. The first type of region is determined as the candidate missed job region.

[0028] The semantic segmentation model includes, but is not limited to, U-Net and SegNet models. U-Net and SegNet are two classic image segmentation network architectures, both employing an encoder-decoder structure. For example, the semantic segmentation model can be deployed in the data processing module 200. After preprocessing the images acquired in real-time by the image acquisition module 100, the data processing module 200 inputs the preprocessed images into the trained semantic segmentation model, outputting a pixel-level segmentation map. This segmentation map contains a first type of region and a second type of region. The first type of region represents the unfinished task area within the mobile device's operational range, while the second type of region represents the completed task area. Based on the segmentation map output by the semantic segmentation model, the first type of region is identified as candidate missed task areas.

[0029] For example, let's take the example of a lawnmower robot performing lawnmowing operations. Figure 3 As shown, after obtaining a high-quality lawn image, it is input into a semantic segmentation model, which outputs a pixel-level segmentation map. Based on the segmentation map, candidate missed areas for the lawn mowing robot are determined.

[0030] In some embodiments, before inputting the image into the trained semantic segmentation model and outputting the segmentation map corresponding to the image, the following steps are included: Obtain training samples, which include labeled images, wherein the labels include a first type of label and a second type of label, the first type of label indicating no task completed and the second type of label indicating a task completed. The training samples are input into the semantic segmentation model to train the semantic segmentation model and obtain the trained semantic segmentation model.

[0031] Before calling the semantic segmentation model, it is first trained based on training samples to obtain a trained semantic segmentation model. For example, different regions, such as untouched and touched areas, in a large number of images can be manually labeled to obtain labeled images. The labels on these labeled images include a first type of label and a second type of label, where the first type of label indicates untouched areas and the second type indicates touched areas. Training samples are then generated based on these labeled images. For example, "trimmed areas" and "untrimmed areas" in a lawn image can be precisely labeled manually to generate training samples. These training samples are then input into the semantic segmentation model for training to obtain a trained semantic segmentation model.

[0032] It should be noted that, in addition to the methods listed above for determining candidate missed job regions through semantic segmentation models, other neural network models such as image segmentation models and deep models can also be used to determine candidate missed job regions, and this application does not impose any specific restrictions.

[0033] S103. Obtain the area or perimeter of the candidate missed work area. If the candidate missed work area meets the preset conditions, then determine the candidate missed work area as the first target missed work area.

[0034] It is understandable that there may be one or more candidate missed job areas, and for each identified candidate missed job area, the operation of obtaining the area or perimeter of the candidate missed job area is performed.

[0035] In some embodiments, such as Figure 5 As shown, step S103 may include sub-step S1031 and sub-step S1032.

[0036] S1031. Perform image morphological processing on the candidate missing job region, the image morphological processing including denoising; S1032. Perform connected component analysis on the processed candidate missed job regions to determine the area or perimeter of the candidate missed job regions.

[0037] For the identified candidate missed job regions, image morphology processing is first performed on the candidate missed job regions. This image morphology processing includes, but is not limited to, denoising. Denoising makes the image clearer while preserving the edge and detail information of the candidate missed job regions, thus providing a more reliable basis for subsequent analysis and processing.

[0038] After performing image morphological processing, connected component analysis is performed on the candidate missed job regions. Each connected component obtained is used as an updated candidate missed job region, and the area or perimeter of the candidate missed job region is determined.

[0039] For example, if the identified candidate missed work areas include multiple regions, during connected component analysis, the adjacency of the multiple candidate missed work areas is first determined. If the distance between two candidate missed work areas is less than or equal to a preset distance threshold, then the two candidate missed work areas are determined to be adjacent; conversely, if the distance between two candidate missed work areas is greater than the preset distance threshold, then the two candidate missed work areas are determined to be non-adjacent. The preset distance threshold can be flexibly set according to actual conditions and is not specifically limited in this application. For example, taking a lawnmower robot as an example, the preset distance threshold can be set to the cutting width of the lawnmower robot. If the distance between two candidate missed work areas is greater than the cutting width of the lawnmower robot, then the two candidate missed work areas are determined to be non-adjacent; if the distance between two candidate missed work areas is less than the cutting width of the lawnmower robot, then the two candidate missed work areas are determined to be adjacent.

[0040] If at least two candidate missed job regions are adjacent, then a connected component analysis is performed on these two regions, and the resulting connected component is used as the updated candidate missed job region to determine its area or perimeter. If at least two candidate missed job regions are not adjacent, then no connected component analysis is performed on them, thus avoiding invalid processing and improving efficiency.

[0041] Based on the area or perimeter of the candidate missed work area, it is determined whether the candidate missed work area meets preset conditions. If the candidate missed work area meets the preset conditions, it is determined as the first target missed work area; otherwise, if the candidate missed work area does not meet the preset conditions, it is determined as the second target missed work area. The first target missed work area is larger than the second target missed work area; that is, the second target missed work area is a smaller area than the first target missed work area.

[0042] For example, the candidate missed operation area satisfies the preset conditions including: The area of ​​the candidate missed work area is greater than or equal to a preset area threshold; or The perimeter of the candidate missed operation area is greater than or equal to a preset perimeter threshold.

[0043] In one implementation, area is used as a consideration. A preset area threshold is set to determine whether a candidate missed work area is a first target missed work area. The specific value of this preset area threshold can be flexibly set according to actual conditions and is not limited in this application. After obtaining the area of ​​the candidate missed work area, the area of ​​the candidate missed work area is compared with the preset area threshold. If the area of ​​the candidate missed work area is greater than or equal to the preset area threshold, the candidate missed work area meets the preset condition, and in this case, the candidate missed work area is determined to be the first target missed work area. If the area of ​​the candidate missed work area is less than the preset area threshold, the candidate missed work area does not meet the preset condition, and in this case, the candidate missed work area is determined to be the second target missed work area.

[0044] In another implementation, perimeter is used as a consideration. A preset perimeter threshold is set to determine whether a candidate missed work area is a first target missed work area. The specific value of this preset perimeter threshold can be flexibly set according to actual conditions and is not limited in this application. After obtaining the perimeter of the candidate missed work area, the perimeter of the candidate missed work area is compared with the preset perimeter threshold. If the perimeter of the candidate missed work area is greater than or equal to the preset perimeter threshold, the candidate missed work area meets the preset condition, and in this case, the candidate missed work area is determined to be the first target missed work area. If the perimeter of the candidate missed work area is less than the preset perimeter threshold, the candidate missed work area does not meet the preset condition, and in this case, the candidate missed work area is determined to be the second target missed work area.

[0045] S104. During the mobile operation, supplementary operations are performed based on the first target missed operation area.

[0046] For the identified first target missed work area, a real-time supplementary work can be initiated immediately. This means pausing the current movement of the mobile device to supplement the missed work area, and then resuming the mobile device's movement after completion. For example, taking a lawnmower robot performing lawnmowing as an example... Figure 3 As shown, after identifying candidate missed areas for the lawnmower robot, it determines whether these areas meet preset conditions to identify the first missed area. If so, the mowing operation based on the pre-planned original path is paused, and the first missed area is re-mown in real time. After the re-mowing is completed, the mowing operation continues. By re-mowing large areas of the first missed area in real time, large-scale missed areas are avoided, thus preventing a negative impact on the user's experience.

[0047] For the identified second target missed work areas, supplementary work is also performed. For example, supplementary work is performed on the second target missed work areas only after the current mobile device's current movement task is completed. Taking a lawnmower robot performing lawnmowing as an example again... Figure 3As shown, if the candidate missed area does not meet the preset conditions, it is identified as the second missed area, and the mowing operation continues based on the pre-planned original path. After the mowing operation is completed, the path is planned based on the second missed area, and the small area of ​​the second missed area is uniformly mowed to improve the operation efficiency.

[0048] In some embodiments, such as Figure 6 As shown, step S104 may include sub-step S1041 and sub-step S1042.

[0049] S1041. Obtain the location information of the first target missed work area; S1042. Perform path planning based on the location information, and perform supplementary operation processing based on the planned first path.

[0050] The location information of the first target missed work area can be determined based on one or more points within the first target missed work area. For example, the location information of the first target missed work area can be determined based on its centroid. For instance, the location information of the first target missed work area includes its position coordinates in the world coordinate system. The location information of the second target missed work area can be determined in the same way.

[0051] Then, based on the location coordinates of the first target missed work area in the world coordinate system, path planning is performed, and supplementary work is carried out based on the planned first path.

[0052] For example, such as Figure 2 As shown, the mobile device 1000 also includes a navigation control module 300 and a storage module 400. The navigation control module 300 includes a GPS (Global Positioning System) / RTK (Real-time kinematic) positioning unit, an IMU (Inertial Measurement Unit), a motor drive controller, etc. The navigation control module 300 is responsible for the basic path planning and motion control of the mobile device 1000 and receives instructions from the data processing module 200, such as the instruction to "proceed to coordinate position a to perform supplementary work". The storage module 400 is used to store images, a trained semantic segmentation model, maps of the first target's missed work area, and maps of the second target's missed work area.

[0053] In some embodiments, such as Figure 7 As shown, step S1041 may include sub-step S10411 and sub-step S10412.

[0054] S10411. Obtain the centroid coordinates of the first target leaked work area, wherein the centroid coordinates are coordinates in the image coordinate system; S10412. Transform the centroid coordinates from the image coordinate system to the world coordinate system to obtain the position coordinates.

[0055] For the identified first target missed work area, firstly, obtain the centroid coordinates of the first target missed work area in the image coordinate system. The centroid coordinates in the image coordinate system are two-dimensional coordinates. Then, perform a coordinate system transformation to convert the centroid coordinates from the image coordinate system to the world coordinate system, obtaining the centroid coordinates in the world coordinate system. The centroid coordinates in the world coordinate system are three-dimensional coordinates. The centroid coordinates in the world coordinate system are determined as the position coordinates of the first target missed work area. For the identified second target missed work area, the position coordinates of the second target missed work area can be determined in the same way.

[0056] In some embodiments, the step of performing path planning based on the location information and performing supplementary task processing based on the planned first path includes: During the mobile operation based on the pre-planned path, a path is planned from the current location of the mobile device to the first target missed operation area according to the location information, the first path is generated, and supplementary operation is performed based on the first path during the mobile operation.

[0057] The currently identified first target missed work areas may include one or more. If only one first target missed work area is identified, a first path can be planned from the current location of the mobile device to this first target missed work area. If multiple first target missed work areas are identified, a first path (e.g., the shortest path from the current location of the mobile device to each first target missed work area) is planned based on the current location of the mobile device and the location information corresponding to the multiple first target missed work areas. The mobile device pauses its work and performs supplementary work processing based on the planned first path, starting from its current location. Once the supplementary work is completed, the paused work resumes execution.

[0058] In some embodiments, for multiple first target missed work areas, some missed work areas can be processed for supplementary work during the movement operation, and others can be processed for supplementary work after the movement operation is completed. For example, the width of the first target missed work area can be obtained. If the width of the first target missed work area is greater than or equal to a preset width threshold, the movement operation is paused, and a first path is planned from the current location of the mobile device to the first target missed work area. The device moves to the first target missed work area based on the first path to perform supplementary work, and then the movement operation continues. If the width of the first target missed work area is less than the preset width threshold, supplementary work is temporarily not performed. After the mobile device completes the movement operation based on the pre-planned path, all first target missed work areas with widths less than the preset width threshold are processed for supplementary work, thereby improving work efficiency. It should be noted that the preset width threshold can be flexibly set according to actual conditions and is not limited in this application.

[0059] For example, taking a lawnmower robot performing lawn mowing as an example, the preset width threshold is set as the cutting width of the lawnmower robot. If the identified target missed mowing areas include areas A, B, C, and D, where the width of area A is greater than the cutting width of the lawnmower robot, and the widths of areas B, C, and D are less than the cutting width of the lawnmower robot, then the lawnmower operation is paused, and a re-mowing path is planned from the current position of the lawnmower robot to area A. Based on the re-mowing path, the robot moves to area A to perform re-mowing. After completion, the lawnmower operation continues. After the lawnmower operation is completed, a re-mowing path is planned to reach areas B, C, and D in sequence. The lawnmower robot moves based on this re-mowing path to perform re-mowing of areas B, C, and D.

[0060] In some embodiments, supplementary work processing is performed based on the second target missed work area, including: After the mobile device completes its mobile operation based on the pre-planned path, a path is planned to pass through all the second target missed operation areas according to the location information corresponding to all the second target missed operation areas, and a second path is generated. Supplementary operation is then performed based on the second path after the mobile operation is completed.

[0061] The mobile device first completes its movement operation based on a pre-planned path. After the movement operation is completed, a second path (e.g., the shortest path through the multiple missed work areas) is planned based on the location information of the missed work areas. The mobile device then moves along the planned second path, sequentially reaching each missed work area to perform supplementary work. By uniformly processing supplementary work in multiple small missed work areas, operational efficiency is improved.

[0062] In the above embodiments, by acquiring an image corresponding to the work area of ​​the mobile device performing mobile operations, a preliminary determination of the missed work area is made based on the image, a candidate missed work area is determined, and the area or perimeter of the candidate missed work area is obtained. If the candidate missed work area meets the preset conditions, the candidate missed work area is determined as the first target missed work area. Then, supplementary work processing is performed based on the first target missed work area, thereby solving the missed work problem and improving the reliability of the mobile device performing mobile operations.

[0063] Please see Figure 8 , Figure 8 This is a structural schematic diagram of a robot provided in an embodiment of this application. The robot includes, but is not limited to, lawnmower robots, indoor mobile robots, patrol robots, etc. Figure 8 As shown, the robot 2000 includes a walking device 500, an image acquisition device 600, and a control device (not shown). The walking device 500 includes, but is not limited to, drive wheels and guide wheels, and is used to control the movement of the robot 2000. The image acquisition device 600 includes, but is not limited to, a camera, a webcam, and a lidar sensor, and acquires images corresponding to the work area where the robot 2000 performs its movement operations. Based on the acquired images, the control device determines a first target missed work area for the robot 2000's movement operations and performs supplementary work based on this first target missed work area.

[0064] For example, a schematic block diagram of the control device is shown below. Figure 9 As shown, the control device 700 may include a processor 710 and a memory 720, wherein the processor 710 and the memory 720 are connected via a bus, such as an I2C (Inter-integrated Circuit) bus.

[0065] Specifically, the processor 710 can be a microcontroller unit (MCU), a central processing unit (CPU), or a digital signal processor (DSP), etc.

[0066] Specifically, the memory 720 can be a Flash chip, a read-only memory (ROM) disk, an optical disk, a USB flash drive, or a portable hard drive, etc. The memory 720 stores various computer programs for the processor 710 to execute.

[0067] The processor 710 is used to run a computer program stored in the memory, and to implement the following when executing the computer program: Acquire images of the work area corresponding to the mobile operation performed by the mobile device; Based on the image, a preliminary determination of the missed work area is made to identify candidate missed work areas; Obtain the area or perimeter of the candidate missed work area. If the candidate missed work area meets the preset conditions, then determine the candidate missed work area as the first target missed work area. During the mobile operation, supplementary operations are performed based on the first target missed operation area.

[0068] In some embodiments, after acquiring the area or perimeter of the candidate missed job region, the processor 710 is configured to: If the candidate missed job area does not meet the preset conditions, then the candidate missed job area is determined as the second target missed job area; After the mobile operation is completed, supplementary operations are performed based on the second target missed operation area.

[0069] In some embodiments, when the processor 710 performs the preliminary determination of the missed job region based on the image and determines the candidate missed job region, it is configured to: The image is input into a trained semantic segmentation model, which outputs a segmentation map corresponding to the image. The segmentation map includes a first type of region and a second type of region. The first type of region represents the unfinished region within the mobile device's operational range, and the second type of region represents the completed region. The first type of region is identified as the candidate missed job region.

[0070] In some embodiments, before implementing the process of inputting the image into the trained semantic segmentation model and outputting the segmentation map corresponding to the image, the processor 710 is configured to perform the following: Obtain training samples, which include labeled images, wherein the labels include a first type of label and a second type of label, the first type of label indicating no task completed and the second type of label indicating a task completed. The training samples are input into the semantic segmentation model to train the semantic segmentation model and obtain the trained semantic segmentation model.

[0071] In some embodiments, the candidate missed job regions satisfying the preset conditions include: The area of ​​the candidate missed work area is greater than or equal to a preset area threshold; or The perimeter of the candidate missed operation area is greater than or equal to a preset perimeter threshold.

[0072] In some embodiments, when the processor 710 acquires the area or perimeter of the candidate missed job region, it is configured to: The candidate missing job regions are subjected to image morphological processing, which includes denoising. Perform connected component analysis on the processed candidate missed job regions to determine the area or perimeter of the candidate missed job regions.

[0073] In some embodiments, when implementing the job replenishment process based on the first target missed job area, the processor 710 is configured to: Obtain the location information of the first target missed work area; Path planning is performed based on the location information, and supplementary tasks are processed based on the planned first path.

[0074] In some embodiments, when the processor 710 implements the location information including position coordinates in the world coordinate system, and acquires the location information of the first target missed work area, it is configured to: Obtain the centroid coordinates of the first target missed work area, where the centroid coordinates are coordinates in the image coordinate system; The centroid coordinates are transformed from the image coordinate system to the world coordinate system to obtain the position coordinates.

[0075] In some embodiments, when the processor 710 performs path planning based on the location information and performs supplementary job processing based on the planned first path, it is configured to: During the mobile operation based on the pre-planned path, a path is planned from the current location of the mobile device to the first target missed operation area according to the location information, the first path is generated, and supplementary operation is performed based on the first path during the mobile operation.

[0076] In some embodiments, when implementing the work replenishment process based on the second target missed work area, the processor 710 is configured to: After the mobile device completes its mobile operation based on the pre-planned path, a path is planned to pass through all the second target missed operation areas according to the location information corresponding to all the second target missed operation areas, and a second path is generated. Supplementary operation is then performed based on the second path after the mobile operation is completed.

[0077] Robot 1000 can execute the missed work area detection method for mobile devices provided in the embodiments of this application. Therefore, it can achieve the beneficial effects that the missed work area detection method for mobile devices provided in the embodiments of this application can achieve. For details, please refer to the previous embodiments, which will not be repeated here.

[0078] For example, taking a lawnmower robot as an example, the robot is equipped with a high-definition wide-angle camera at its front end. This camera captures images of the lawn, which undergo preprocessing such as distortion correction, color correction, and illumination equalization to eliminate environmental interference and obtain high-quality images. The lawn images are then input into a semantic segmentation model, which outputs pixel-level segmentation maps. Based on these maps, candidate missed-mowing regions are identified. Whether a candidate missed-mowing region meets preset conditions is determined to be a first-target missed-mowing region. For instance, the model checks if the area of ​​the candidate missed-mowing region is greater than or equal to a preset area threshold. If the area is greater than or equal to the threshold, the region meets the preset conditions and is designated as the first-target missed-mowing region. If the area is less than the threshold, the region does not meet the preset conditions and is designated as a second-target missed-mowing region. If the area is identified as the second target missed area, the mowing operation continues based on the pre-planned path. After the mowing operation is completed, the missed area of ​​the second target is re-mown. If the area is the first target missed area, the missed area of ​​the first target is re-mown. After the re-mowing is completed, the mowing operation continues, thereby solving the problem of missed lawn mowing and improving the reliability of the mowing robot in mowing operations.

[0079] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for detecting missing work areas in a removable device.

[0080] The computer-readable storage medium may be an internal storage unit of the control device or robot described in the foregoing embodiments, such as a hard disk or memory of the control device or robot. Alternatively, the computer-readable storage medium may be an external storage device of the control device or robot, such as a plug-in hard disk, SmartMediaCard (SMC), SecureDigitalCard (SDCard), or FlashCard.

[0081] Since the computer program stored in the storage medium can execute any of the missed work area detection methods for mobile devices provided in the embodiments of this application, the beneficial effects that any of the missed work area detection methods for mobile devices provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0082] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0083] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the scope of the technology disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application.

Claims

1. A method for detecting missed work areas in a mobile device, characterized in that, include: Acquire images of the work area corresponding to the mobile operation performed by the mobile device; Based on the image, a preliminary determination of the missed work area is made to identify candidate missed work areas; Obtain the area or perimeter of the candidate missed work area. If the candidate missed work area meets the preset conditions, then determine the candidate missed work area as the first target missed work area. During the mobile operation, supplementary operations are performed based on the first target missed operation area.

2. The method for detecting missed work areas in a mobile device according to claim 1, characterized in that, After obtaining the area or perimeter of the candidate missed job region, the process includes: If the candidate missed job area does not meet the preset conditions, then the candidate missed job area is determined as the second target missed job area; After the mobile operation is completed, supplementary operations are performed based on the second target missed operation area.

3. The method for detecting missed work areas in a mobile device according to claim 1, characterized in that, The preliminary determination of missed job areas based on the image, and the identification of candidate missed job areas, includes: The image is input into a trained semantic segmentation model, which outputs a segmentation map corresponding to the image. The segmentation map includes a first type of region and a second type of region. The first type of region represents the unfinished region within the mobile device's operational range, and the second type of region represents the completed region. The first type of region is identified as the candidate missed job region.

4. The method for detecting missed work areas in a mobile device according to claim 3, characterized in that, Before inputting the image into the trained semantic segmentation model and outputting the segmentation map corresponding to the image, the process includes: Obtain training samples, which include labeled images, wherein the labels include a first type of label and a second type of label, the first type of label indicating no task completed and the second type of label indicating a task completed. The training samples are input into the semantic segmentation model to train the semantic segmentation model and obtain the trained semantic segmentation model.

5. The method for detecting missed work areas in a mobile device according to claim 1, characterized in that, The candidate missed operation areas satisfy the preset conditions including: The area of ​​the candidate missed work area is greater than or equal to a preset area threshold; or The perimeter of the candidate missed operation area is greater than or equal to a preset perimeter threshold.

6. The method for detecting missed work areas in a mobile device according to claim 1, characterized in that, Obtaining the area or perimeter of the candidate missed job region includes: The candidate missing job regions are subjected to image morphological processing, which includes denoising. Perform connected component analysis on the processed candidate missed job regions to determine the area or perimeter of the candidate missed job regions.

7. The method for detecting missed work areas in a mobile device according to claim 1, characterized in that, The supplementary work processing based on the first target missed work area includes: Obtain the location information of the first target missed work area; Path planning is performed based on the location information, and supplementary tasks are processed based on the planned first path.

8. The method for detecting missed work areas in a mobile device according to claim 7, characterized in that, The location information includes location coordinates in the world coordinate system, and obtaining the location information of the first target missed work area includes: Obtain the centroid coordinates of the first target missed work area, where the centroid coordinates are coordinates in the image coordinate system; The centroid coordinates are transformed from the image coordinate system to the world coordinate system to obtain the position coordinates.

9. The method for detecting missed work areas in a mobile device according to claim 7, characterized in that, The step of performing path planning based on the location information and performing supplementary work processing based on the planned first path includes: During the mobile operation based on the pre-planned path, a path is planned from the current location of the mobile device to the first target missed operation area according to the location information, the first path is generated, and supplementary operation is performed based on the first path during the mobile operation.

10. The method for detecting missed work areas in a mobile device according to claim 2, characterized in that, The supplementary work processing based on the second target missed work area includes: After the mobile device completes its mobile operation based on the pre-planned path, a path is planned to pass through all the second target missed operation areas according to the location information corresponding to all the second target missed operation areas, and a second path is generated. Supplementary operation is then performed based on the second path after the mobile operation is completed.

11. A robot, characterized in that, The robot includes a walking device, an image acquisition device, and a control device; wherein, the walking device is used to move the robot; the image acquisition device is used to acquire images corresponding to the working area where the robot performs the moving operation; the control device includes a processor and a memory, the memory storing a computer program that can be executed by the processor, and when the computer program is executed by the processor, it implements the steps of the method for detecting missing working areas of a mobile device as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the method for detecting missing work areas of a mobile device as described in any one of claims 1 to 10.

Images