Self-moving cleaning equipment control method and equipment

Abstract

The invention provides a self-moving cleaning equipment control method and equipment, and relates to the field of intelligent robots. The method comprises the steps that in the process that the self-moving cleaning equipment executes a cleaning task, after an obstacle is recognized and determined as a wire harness obstacle, a wire harness obstacle image is obtained; determining position coordinates of the wiring harness in the image, and avoiding a wiring harness obstacle based on the position coordinates; determining the position coordinates of the wire harness in the image: determining a wire harness adjacent area of the image, determining a connecting line between the position coordinates of each pixel point in the wire harness adjacent area and the position coordinates of the wire harness obstacle, and determining the position coordinates of the wire harness obstacle based on the connecting line; the area adjacent to the wire harness at least comprises an enveloping area of a wire harness obstacle; thus, accurate position calculation is carried out through the connecting wire, the accuracy of wire harness obstacle position recognition is improved, faults and damage of equipment caused by wire harness winding or pulling are reduced by avoiding wire harness obstacles, and the safety and reliability of the equipment are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent robots, and in particular to a control method and device for a self-moving cleaning device. Background Art

[0002] In the field of automation and robotics, especially in the application of self-propelled robots such as sweeping robots, identifying and avoiding obstacles is one of the important tasks to ensure the safe operation of the equipment. Wire harness obstacles, such as power cables, network cables, charging cables, etc., have irregular shapes and may be distributed in different directions and positions. If they are not accurately identified, they may be drawn into the driving wheels of the sweeping robot during the cleaning process, affecting the operation of the sweeping robot.

[0003] In the prior art, self-propelled robots usually use box detection algorithms to identify harness obstacles. Box detection algorithms use rectangular boxes to represent the position and range of objects. However, harnesses are usually slender, curved, and may be arbitrarily distributed in space. Using rectangular boxes to represent the position of the harness has low positioning accuracy.

[0004] However, due to the complex shape of the wire harness, different labelers may give different labels to the same wire harness. This inconsistency affects the detection results of the algorithm, which in turn causes the self-propelled robot to reduce the recognition accuracy when detecting the position of the wire harness. Summary of the invention

[0005] The present application provides a self-moving cleaning device control method and device, which can effectively identify the position of the wire beam obstacle by considering the connection line between each pixel point and the wire beam obstacle in the wire beam vicinity area and accurately locating the wire beam obstacle based on the connection line.

[0006] In a first aspect, the present application provides a self-moving cleaning device control method, which is applied to the self-moving cleaning device, and the method comprises:

[0007] During the process of the self-moving cleaning device performing a cleaning task, after identifying an obstacle and determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle is acquired;

[0008] Determine the position coordinates of the wire harness in the wire harness obstacle image, and avoid the wire harness obstacle based on the position coordinates;

[0009] Wherein, determining the position coordinates of the wire beam in the wire beam obstacle image includes:

[0010] Determine a wire beam neighboring area of ​​the wire beam obstacle image, determine a connecting line between the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and determine the position coordinates of the wire beam obstacle based on the connecting line; the wire beam neighboring area at least includes an envelope area of ​​the wire beam obstacle; the envelope area is a closed area created around the boundary of the wire beam obstacle area.

[0011] In this way, when the self-mobile cleaning device is performing cleaning tasks, the position of the wiring harness obstacle can be accurately determined by detailed analysis of the connection between the pixel points and the wiring harness obstacles. After the position coordinates are determined, the path can be planned more effectively to avoid potential collisions or interferences, thereby reducing the physical contact between the device and the wiring harness, reducing the risk of entanglement, damage to the wiring harness or the device itself, improving safety, and completing the cleaning task more efficiently.

[0012] Optionally, the position coordinates of the harness obstacle are determined based on the connection line, including:

[0013] The position coordinates of the wire beam obstacle are determined based on the distance of the connecting line; wherein the distance of each connecting line is the minimum value of the distance between the position coordinates of the corresponding pixel point in the wire beam adjacent area and the connecting line of the position coordinates of the wire beam obstacle.

[0014] In this way, since the distance from each pixel point in the adjacent area of ​​the wire beam to the wire beam obstacle is taken into account, and by calculating and comparing the minimum distance value, the wire beam obstacle can be accurately located, and then the position of the wire beam obstacle can be effectively identified, especially in complex environments, where the wire beam obstacles may have irregular shapes, and the distance calculation method based on the connection line can adapt to these complexities and provide more reliable positioning results.

[0015] Optionally, determining a line beam neighboring area of ​​the line beam obstacle image includes:

[0016] Obtain a predefined neighborhood window, and calculate the neighborhood features of each pixel point in the beam obstacle image within the predefined neighborhood window;

[0017] According to the neighborhood features, the pixel points in the beam obstacle image are segmented by threshold value to obtain the beam neighboring area and the non-beam neighboring area;

[0018] Among them, the self-moving cleaning device does not process areas that are not adjacent to the wiring harness.

[0019] In this way, the processing rate can be improved by identifying and ignoring non-wiring harness adjacent areas, and by determining the wiring harness adjacent areas and non-wiring harness adjacent areas through the above-mentioned neighborhood segmentation method, the wiring harness adjacent areas that need attention can be more accurately determined, thereby improving the subsequent recognition accuracy of wiring harness obstacles.

[0020] Optionally, determining the position coordinates of the wire harness in the wire harness obstacle image includes:

[0021] The wire harness obstacle image is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the wire harness obstacle image.

[0022] In this way, the trained wire harness detection model is used to predict the position coordinates of the wire harness obstacles, which can identify the wire harness obstacles with high precision in complex backgrounds, adapt to various environmental changes, and have strong robustness. In addition, the above-mentioned wire harness detection model can also adapt to new environments or identify various types of wire harness obstacles through transfer learning or further training, and has good scalability.

[0023] Optionally, the wire harness detection model includes a classification model and a regression model; the wire harness obstacle image is input into the trained wire harness detection model, and the position coordinates of the wire harness obstacle in the wire harness obstacle image are predicted, including:

[0024] Input the wire harness obstacle image into the trained classification model to obtain the wire harness adjacent area in the wire harness obstacle image;

[0025] The adjacent area of ​​the wire harness is input into the trained regression model to obtain the position coordinates of the wire harness obstacles in the adjacent area of ​​the wire harness.

[0026] In this way, the present application can optimize the performance of each model separately and improve the overall detection accuracy by decomposing the model prediction into two steps: classification and regression. The classification model first narrows the scope of the area that needs to be accurately positioned, reduces the computational burden of the regression model, and improves processing efficiency. The trained regression model can then be used to perform accurate regression positioning, fit the irregular shapes of the wiring harness obstacles, identify and accurately locate the wiring harness, and then output the specific position of the wiring harness.

[0027] Optionally, the wire harness detection model is a neural network model based on deep learning.

[0028] It should be noted that the neural network model based on deep learning performs well in image recognition and classification tasks, can process complex image features, and provide high-precision detection results. In addition, the neural network model based on deep learning can effectively identify target objects in complex and changing backgrounds, and is suitable for wire harness detection in various environments and conditions. In particular, the model is highly robust to noise, occlusion, and illumination changes, and can adapt to different image qualities and environmental conditions.

[0029] In this way, the deep learning-based neural network model can identify the wiring harness and accurately locate its specific position.

[0030] Optionally, the training process of the harness detection model includes:

[0031] Acquire a training data set; the training data set includes a plurality of training samples, each training sample includes a wire harness obstacle image and position coordinates of the wire harness obstacle in the wire harness obstacle image;

[0032] The training data set is input into the wire harness detection model for training to obtain a trained wire harness detection model;

[0033] The training process includes: determining the wire beam neighboring area of ​​each wire beam obstacle image, and determining the minimum distance corresponding to the line connecting the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and performing regression training using the line corresponding to the minimum distance, the position coordinates of the pixel points corresponding to each line, and the position coordinates of the wire beam obstacle.

[0034] In this way, by calculating the minimum distance and performing regression training, the model can accurately locate the position of the wiring harness obstacle and improve the detection accuracy. Furthermore, through precise regression training, the model can reduce the error in position prediction and improve the overall reliability. Among them, the wiring harness detection model can automatically learn the complex features in the image, reducing the dependence on manual work.

[0035] Optionally, the wire harness detection model includes a classification model and a regression model; the training data set is input into the wire harness detection model for training to obtain a trained wire harness detection model, including:

[0036] For each wire beam obstacle image, obtaining a wire beam neighboring area in the labeled wire beam obstacle image;

[0037] Inputting a plurality of wire beam obstacle images and a wire beam neighboring region corresponding to each wire beam obstacle image into a classification model for training to obtain a trained classification model;

[0038] Inputting the position coordinates of the plurality of wire harness adjacent regions and the wire harness obstacles in the wire harness adjacent regions into the regression model for training, thereby obtaining a trained regression model;

[0039] The trained classification model and the trained regression model are aggregated to obtain a trained wire harness detection model.

[0040] In this way, by combining the classification model and the regression model, the possible adjacent areas of the wiring harness can be identified, and then the position of the wiring harness can be accurately located to improve the overall detection accuracy. The independent training and optimization of the classification model and the regression model make the overall design more modular and easy to maintain and expand. Among them, the classification model first narrows the scope of the area that needs to be accurately processed, effectively filters out irrelevant areas, reduces the computational burden of the regression model, and improves processing efficiency. Then the regression model accurately locates the wiring harness obstacles and improves positioning accuracy.

[0041] Optionally, the training process of the regression model includes:

[0042] Obtain the position coordinates of each pixel point in the adjacent area of ​​the line beam;

[0043] For each pixel point, using the position coordinates of the pixel point and the position coordinates of the beam obstacle, multiple distances from the pixel point to multiple pixel points corresponding to the beam obstacle are calculated respectively;

[0044] Determine a minimum distance value among multiple distances, and determine a connection line between pixel points corresponding to the minimum distance value;

[0045] Regression training is performed by determining the connecting lines between the pixel points corresponding to the minimum distance, the position coordinates of the pixel points corresponding to each connecting line, and the position coordinates of the harness obstacles.

[0046] In this way, by utilizing the minimum distance between pixel points, the model can accurately capture the position of the harness obstacles and improve positioning accuracy. The minimum distance is determined by precise distance calculation and connection, so that the model can reduce errors in position prediction and improve overall reliability.

[0047] Optionally, avoiding obstacles in the wiring harness based on the position coordinates includes at least one of the following situations:

[0048] The cleaning route is replanned based on the position coordinates of the wiring harness obstacles, and the self-moving cleaning device performs the cleaning task according to the cleaning route to avoid the wiring harness obstacles;

[0049] The cleaning restricted area is determined based on the position coordinates of the harness obstacle. After the self-mobile cleaning equipment cleans around the cleaning restricted area, it continues to perform the cleaning task according to the originally planned path.

[0050] The position coordinates of the wire harness obstacle are sent to the user terminal for display, and the wire harness obstacle is avoided in response to the user's control operation.

[0051] In this way, after the self-mobile cleaning device determines the position coordinates of the wire harness obstacle, it can avoid the wire harness obstacle in a variety of ways to adapt to different environmental changes and user needs. While improving application flexibility, it can also reduce the risk of collision between the device and the wire harness obstacle, thereby protecting the equipment and other items in the environment.

[0052] In a second aspect, the present application provides a method for controlling a self-moving cleaning device, which is applied to a user terminal, and the method includes:

[0053] receiving the position coordinates of the wire harness in the wire harness obstacle image determined during the cleaning task sent by the mobile cleaning device, and visually displaying the position coordinates;

[0054] Wherein, determining the position coordinates of the wire beam in the wire beam obstacle image includes:

[0055] After identifying the obstacle and determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle is obtained;

[0056] Determine a wire beam neighboring area of ​​the wire beam obstacle image, determine a connecting line between the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and determine the position coordinates of the wire beam obstacle based on the connecting line; the wire beam neighboring area at least includes an envelope area of ​​the wire beam obstacle; the envelope area is a closed area created around the boundary of the wire beam obstacle area.

[0057] It can be understood that by displaying the location of the harness obstacle on the user terminal, the user can take preventive measures to avoid the collision between the self-moving cleaning device and the harness obstacle, thereby protecting the self-moving cleaning device and the surrounding environment.

[0058] In this way, by providing real-time visual content and interactive functions, the opportunities for users to participate in equipment operation are increased, user experience and satisfaction are improved, and users can dynamically adjust the cleaning strategy of the equipment according to the real-time displayed visual content, more accurately control the behavior of the equipment, and reduce the possibility of misoperation, thereby reducing the risk of damage to the self-mobile cleaning equipment.

[0059] In a third aspect, the present application provides a self-moving cleaning device, the self-moving cleaning device comprising: a processor, and a memory communicatively connected to the processor;

[0060] Memory stores computer-executable instructions;

[0061] The processor executes the computer-executable instructions stored in the memory to implement any method as described in the first aspect.

[0062] In summary, the present application provides a method and device for controlling a self-moving cleaning device. In the process of the self-moving cleaning device performing a cleaning task, after identifying a wiring harness obstacle, an image of the wiring harness obstacle is acquired to identify a wiring harness proximity area of ​​the wiring harness in the image. The wiring harness proximity area at least includes an envelope area of ​​the wiring harness obstacle. The envelope area is a closed area created around the boundary of the wiring harness obstacle. Further, a line connecting the position coordinates of each pixel point in the wiring harness proximity area and the position coordinates of the wiring harness obstacle is found. The line has a common intersection with the wiring harness obstacle. These common intersections are The same intersection points can better fit the non-fixed shape of the wiring harness. Furthermore, based on these connecting lines, the position coordinates of the wiring harness obstacles can be accurately determined, and then the self-moving cleaning equipment can adjust its path planning using the determined wiring harness position coordinates to avoid wiring harness obstacles and ensure the smooth progress of the cleaning task. In this way, accurate image analysis and position calculation are performed through the connecting lines to improve the accuracy of wiring harness obstacle position identification, and by accurately identifying and avoiding wiring harness obstacles, the failure and damage of the self-moving cleaning equipment caused by wiring harness entanglement or pulling is reduced, thereby improving the safety and reliability of the equipment. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0064] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0065] Figure 2 A schematic diagram of a flow chart of a method for controlling a self-moving cleaning device provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of a scenario of interaction between a user terminal and a self-moving cleaning device provided in an embodiment of the present application;

[0067] Figure 4 A schematic diagram of the structure of a self-moving cleaning equipment control device provided in an embodiment of the present application;

[0068] Figure 5 A schematic structural diagram of a self-moving cleaning device provided in an embodiment of the present application.

[0069] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0070] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0071] In order to facilitate the clear description of the technical solutions of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish the same items or similar items with substantially the same functions and effects. For example, the first device and the second device are only used to distinguish different devices, and their order is not limited. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not necessarily limit them to be different.

[0072] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0073] In the present application, "at least one" means one or more, and "plurality" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0074] A self-moving cleaning device is a self-moving robot specially designed for cleaning tasks, such as a sweeping robot and an automatic floor scrubber. The self-moving cleaning device is used to sweep the surface to be cleaned, vacuum, wet mop, etc. to keep the environment clean.

[0075] In one possible implementation, the self-propelled cleaning device can use a box detection algorithm to identify harness obstacles and avoid them during the cleaning process.

[0076] However, the box detection algorithm uses a rectangular box to represent the position and range of an object, so the rectangular box cannot well represent the irregular shape of the wiring harness obstacle, and it is difficult to locate the true position of the wiring harness obstacle, resulting in misjudgment of the self-moving cleaning equipment when avoiding obstacles; and in the labeling stage, there may be ambiguity in how to define the boundaries of the wiring harness obstacle, especially when the wiring harness is staggered or partially blocked; therefore, the box detection algorithm is usually designed for objects with clear boundaries and shapes, and has poor adaptability to wiring harnesses without fixed shapes.

[0077] It is understandable that if the position of the wiring harness obstacle is not accurately identified during the cleaning process of the self-mobile cleaning equipment, the wiring harness obstacle may be entangled in the driving wheel of the self-mobile cleaning equipment, causing the self-mobile cleaning equipment to stop working or be damaged; the self-mobile cleaning equipment may also pull the wiring harness obstacle, causing items on the table and chairs to fall, causing losses to the user and affecting the user experience.

[0078] In another possible implementation, the self-moving cleaning device can identify objects and terrain in the environment through semantic segmentation and perform obstacle avoidance, navigation, path planning, etc.; the semantic segmentation is a deep learning algorithm that associates a label or category with each pixel of an image, and is used to identify a set of pixels that constitute distinguishable categories, each pixel in the pixel set is labeled as a specific category.

[0079] Among them, labeling pixel categories is a time-consuming and expensive process, especially when high precision is required. Therefore, during the cleaning process of the self-moving cleaning equipment, semantic segmentation is used to identify wire harness obstacles. Although the shape and position of the wire harness can be identified, a large amount of pixel-level data needs to be processed, and the computational complexity is high, especially in real-time applications. This may cause delays, which in turn affects the obstacle avoidance effect of the self-moving cleaning equipment.

[0080] In view of the above problems, the present application provides a control method for a self-moving cleaning device. During the process of the self-moving cleaning device performing a cleaning task, after identifying a wiring harness obstacle, an image of the wiring harness obstacle is obtained to identify a wiring harness adjacent area of ​​the wiring harness in the image. The wiring harness adjacent area at least includes an envelope area of ​​the wiring harness obstacle, and the envelope area is a closed area created around the boundary of the wiring harness obstacle. Further, a line connecting the position coordinates of each pixel point in the wiring harness adjacent area and the position coordinates of the wiring harness obstacle is found. This line has a common intersection with the wiring harness obstacle, and these common intersections can better fit the non-fixed shape of the wiring harness. Further, based on these lines, the position coordinates of the wiring harness obstacle are accurately determined, and then using the determined wiring harness position coordinates, the self-moving cleaning device adjusts its path planning to avoid the wiring harness obstacle and ensure the smooth progress of the cleaning task. In this way, accurate image analysis and position calculation are performed through the lines, thereby improving the accuracy of wiring harness obstacle position identification, and by accurately identifying and avoiding wiring harness obstacles, the failure and damage of the self-moving cleaning device caused by wiring harness entanglement or pulling are reduced, thereby improving the safety and reliability of the equipment.

[0081] For example, Figure 1 A schematic diagram of an application scenario provided in an embodiment of the present application, such as Figure 1 As shown, the application scenario is applied to a scenario in which a self-mobile cleaning device performs a cleaning task. Taking a sweeping robot as an example, the application scenario includes: a sweeping robot 100 and a harness obstacle 101. The sweeping robot 100 has a built-in neural network processing unit (NPU) chip, and the NPU chip is used to execute the method of an embodiment of the present application.

[0082] When the sweeping robot 100 performs a cleaning task in the living room, if a harness obstacle 101 is identified, an image of the harness obstacle 101 can be obtained. The image can be an RGB image taken by an RGB (Red, Green, Blue) camera. The embodiment of the present application does not specifically limit the type of image and the shooting device.

[0083] Furthermore, the RGB image is transmitted to the NPU chip of the sweeping robot 100, so that the NPU chip processes the RGB image, determines the position coordinates of the wire harness obstacle 101 in the image, and then avoids the wire harness obstacle 101 based on the position coordinates and continues to perform the cleaning task.

[0084] Part of the wire harness in the wire harness obstacle 101 is located on the surface to be cleaned, and the rest of the wire harness is suspended in the air. Then, the image of the wire harness obstacle 101 taken by the cleaning robot 100 at least includes the image of the part of the wire harness located on the surface to be cleaned.

[0085] Optionally, if the image of the wiring harness obstacle 101 taken by the sweeping robot 100 is an image of a portion of the wiring harness on the surface to be cleaned, the remaining portion of the wiring harness above the sweeping robot 100 does not need to be photographed, because the remaining portion of the wiring harness may not affect the sweeping robot 100 in performing the cleaning task. In this way, the image data processing requirements can be reduced and processing efficiency can be improved.

[0086] Optionally, if the image of the wiring harness obstacle 101 taken by the sweeping robot 100 corresponds to the image of all the wiring harnesses within the scanning field of view of the sweeping robot 100, and the image may include some wiring harnesses that are higher than the sweeping robot 100, the NPU chip can use the projection algorithm to project the wiring harnesses in the image, and then determine the position coordinates of the wiring harness obstacle 101 in the image after the projection processing.

[0087] It should be noted that the self-moving cleaning device control method can also be applied to shopping malls, schools, and offices. The present application does not limit the specific application scenarios of the self-moving cleaning device control method, and the above is only an example.

[0088] For example, Figure 2 A flow chart of a method for controlling a self-moving cleaning device provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the self-moving cleaning device control method is applied to the self-moving cleaning device. The execution subject of the self-moving cleaning device control method may be the NPU chip mentioned above, or may be a processor or controller in the self-moving cleaning device. The embodiment of the present application does not specifically limit this. The method includes the following steps:

[0089] S201. During the process of a self-moving cleaning device performing a cleaning task, after an obstacle is identified and determined to be a wire harness obstacle, an image of the wire harness obstacle is acquired.

[0090] In the present application, during the process of the self-mobile cleaning equipment performing the cleaning task, sensors such as cameras, lidars, infrared sensors, ultrasonic sensors, etc. can be used to scan the surrounding environment to determine whether there are obstacles in the surrounding environment. The embodiments of the present application do not specifically limit the type of sensors for identifying obstacles.

[0091] Among them, after determining that there is an obstacle, the type of obstacle can be identified based on computer vision technology, such as artificial intelligence (AI) algorithm, image processing algorithm, etc. The embodiment of the present application does not specifically limit the method for identifying the type of obstacle.

[0092] Furthermore, after determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle may be acquired for further analysis and processing.

[0093] It should be noted that the embodiment of the present application does not specifically limit the content of the image of the wire harness obstacle obtained by the self-mobile cleaning device, which can be determined based on the functional setting of the self-mobile cleaning device, but the image of the wire harness obstacle at least includes the image of the wire harness located on the surface to be cleaned.

[0094] S202, determining the position coordinates of the wire harness in the wire harness obstacle image, and avoiding the wire harness obstacle based on the position coordinates;

[0095] Wherein, determining the position coordinates of the wire beam in the wire beam obstacle image includes:

[0096] Determine a wire beam neighboring area of ​​the wire beam obstacle image, determine a connecting line between the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and determine the position coordinates of the wire beam obstacle based on the connecting line; the wire beam neighboring area at least includes an envelope area of ​​the wire beam obstacle; the envelope area is a closed area created around the boundary of the wire beam obstacle area.

[0097] In an embodiment of the present application, the envelope area is a closed convex polygon, which includes all points in the point set corresponding to the wiring harness obstacle area and has the smallest area among all convex areas that may contain these point sets. The envelope area is a closed area created around the boundary of the wiring harness obstacle, which can ensure that all possible wiring harness obstacles are taken into consideration.

[0098] In this step, an image containing a harness obstacle is acquired, and a harness neighborhood area of ​​the harness obstacle in the image is determined, the harness neighborhood area at least including an envelope area of ​​the harness obstacle. Further, within the harness neighborhood area, the position coordinates of each pixel point are identified, the position coordinates are usually expressed as pixel positions of the image; further, a connecting line between the position coordinates of each pixel point and the position coordinates of the harness obstacle is determined, and then the position coordinates of the harness obstacle are determined based on the connecting line using a predefined algorithm.

[0099] Among them, the predefined algorithm can be a minimum method, a Euclidean distance algorithm, a Dijkstra algorithm, etc., and the embodiments of the present application do not make specific limitations on this.

[0100] Optionally, the position coordinates of the harness obstacle are determined based on the connection line, including:

[0101] The position coordinates of the wire beam obstacle are determined based on the distance of the connecting line; wherein the distance of each connecting line is the minimum value of the distance between the position coordinates of the corresponding pixel point in the wire beam adjacent area and the connecting line of the position coordinates of the wire beam obstacle.

[0102] For example, for each pixel point, the line between it and the beam obstacle can be calculated, and by analyzing these line distances, it is determined which pixel point or pixels have the smallest line distance between the beam obstacle and the pixel point. These pixel points with the smallest distance are used to help determine the precise position coordinates of the obstacle.

[0103] In this way, since the distance from each pixel point in the adjacent area of ​​the wire beam to the wire beam obstacle is taken into account, and by calculating and comparing the minimum distance value, the wire beam obstacle can be accurately located, and then the position of the wire beam obstacle can be effectively identified, especially in complex environments, where the wire beam obstacles may have irregular shapes, and the distance calculation method based on the connection line can adapt to these complexities and provide more reliable positioning results.

[0104] In the present application, after determining the position coordinates of the wiring harness obstacle, a path can be planned or an adjustment operation can be performed based on the position coordinates of the wiring harness obstacle to avoid the wiring harness obstacle; wherein the adjustment operation may involve adjusting the movement path of the device to ensure that there is no collision with the wiring harness obstacle. The embodiments of the present application do not specifically limit the obstacle avoidance strategy adopted.

[0105] Therefore, when the self-mobile cleaning device is performing cleaning tasks, the position of the wiring harness obstacle can be accurately determined by analyzing the connection between the pixel points and the wiring harness obstacles in detail. After the position coordinates are determined, the path can be planned more effectively to avoid potential collisions or interferences, thereby reducing the physical contact between the device and the wiring harness, reducing the risk of entanglement, damage to the wiring harness or the device itself, improving safety, and completing the cleaning task more efficiently.

[0106] It can be understood that since the above process can be automatically performed by an algorithm, the need for human intervention is reduced, and the above method relies on data calculation rather than visual or other imprecise methods, thereby improving the accuracy and consistency of positioning.

[0107] Optionally, determining a line beam neighboring area of ​​the line beam obstacle image includes:

[0108] Obtain a predefined neighborhood window, and calculate the neighborhood features of each pixel point in the beam obstacle image within the predefined neighborhood window;

[0109] According to the neighborhood features, the pixel points in the beam obstacle image are segmented by threshold value to obtain the beam neighboring area and the non-beam neighboring area;

[0110] Among them, the self-moving cleaning device does not process areas that are not adjacent to the wiring harness.

[0111] It should be noted that the wire harness adjacent area and the non-wire harness adjacent area together constitute the wire harness obstacle image, and these two areas do not overlap; there is no wire harness obstacle in the non-wire harness area, so the non-wire harness area is not processed, thereby optimizing the processing efficiency.

[0112] In the present application, the wiring harness obstacle image is segmented. Different from the conventional segmentation method, since the wiring harness is not a convex polygon and cannot be fitted with a polygon, the NPU chip uses a neighborhood segmentation method for the wiring harness obstacle image to obtain a non-wiring harness area and a wiring harness adjacent area. Among them, since the wiring harness is relatively special and has no regular shape, it is impossible to obtain an accurate area in one step. Therefore, the wiring harness adjacent area is the approximate area of ​​the wiring harness obstacle obtained through neighborhood segmentation.

[0113] In this step, a predefined neighborhood window size is selected to analyze the local features of each pixel in the image. The predefined neighborhood window can be a rectangular or square area of ​​a fixed size, which is not specifically limited in the embodiments of the present application.

[0114] Furthermore, for each pixel in the image, its neighborhood features within a predefined neighborhood window are calculated, which may include information such as color, brightness, texture, etc., and then by analyzing these neighborhood features, it is possible to identify which pixels belong to the wire harness obstacles.

[0115] Furthermore, according to the calculated neighborhood features, the pixel points in the image are subjected to threshold segmentation, that is, an appropriate threshold is selected to divide the pixel points into a line-bundle-adjacent area and a non-line-bundle-adjacent area.

[0116] The selection of the threshold may be determined based on experience, statistical analysis, or machine learning methods to ensure accurate segmentation. The embodiment of the present application does not specifically limit the size of the threshold.

[0117] In this way, the processing rate can be improved by identifying and ignoring non-wiring harness adjacent areas, and by determining the wiring harness adjacent areas and non-wiring harness adjacent areas through the above-mentioned neighborhood segmentation method, the wiring harness adjacent areas that need attention can be more accurately determined, thereby improving the subsequent recognition accuracy of wiring harness obstacles.

[0118] Optionally, in dividing the line beam adjacent area and the non-line beam adjacent area, the semantic segmentation method can also be used to perform pixel-by-pixel segmentation. The boundaries of the two divided sub-image areas are clear, and they are the line beam adjacent area and the non-line beam adjacent area respectively. The embodiment of the present application does not specifically limit the method for dividing the line beam adjacent area and the non-line beam adjacent area.

[0119] Optionally, determining the position coordinates of the wire harness in the wire harness obstacle image includes:

[0120] The wire harness obstacle image is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the wire harness obstacle image.

[0121] In an embodiment of the present application, using a trained wire harness detection model to determine the position coordinates of the wire harness in the wire harness obstacle image is a method based on machine learning. The embodiment of the present application does not limit the specific model type corresponding to the wire harness detection model. For example, the wire harness detection model can be a convolutional neural network, a recurrent neural network model, an autoencoder model, a generative adversarial network model, a variational autoencoder model, etc.

[0122] Exemplarily, the wire harness obstacle image to be processed is input into a trained wire harness detection model, and the model extracts image features through its neural network layer. Further, the model analyzes the image features to predict the position coordinates of the wire harness obstacle.

[0123] Therefore, using the trained wire harness detection model to predict the position coordinates of wire harness obstacles can identify wire harness obstacles with high precision in complex backgrounds, adapt to various environmental changes, and have strong robustness. In addition, the above-mentioned wire harness detection model can also adapt to new environments or identify various types of wire harness obstacles through transfer learning or further training, and has good scalability.

[0124] Optionally, the wire harness detection model is a neural network model based on deep learning.

[0125] For example, Figure 1 As shown, the NPU chip uses a pre-trained neural network model to process the RGB image to identify and locate the exact position of the wire harness; wherein, the neural network model is pre-trained and is trained by learning the annotation results of the wire harness in a large number of images.

[0126] It should be noted that the neural network model based on deep learning performs well in image recognition and classification tasks, can process complex image features, and provide high-precision detection results. In addition, the neural network model based on deep learning can effectively identify target objects in complex and changing backgrounds, and is suitable for wire harness detection in various environments and conditions. In particular, the model is highly robust to noise, occlusion, and illumination changes, and can adapt to different image qualities and environmental conditions.

[0127] In this way, the deep learning-based neural network model can identify the wiring harness and accurately locate its specific position.

[0128] Among them, the neural network model based on deep learning includes a regression processing process to output the position coordinates of the wire harness obstacles. Therefore, the inaccurate wire harness boundary caused by neighborhood segmentation can be offset by precise regression positioning to obtain the accurate position of the wire harness.

[0129] Optionally, the wire harness detection model includes a classification model and a regression model; the wire harness obstacle image is input into the trained wire harness detection model, and the position coordinates of the wire harness obstacle in the wire harness obstacle image are predicted, including:

[0130] Input the wire harness obstacle image into the trained classification model to obtain the wire harness adjacent area in the wire harness obstacle image;

[0131] The adjacent area of ​​the wire harness is input into the trained regression model to obtain the position coordinates of the wire harness obstacles in the adjacent area of ​​the wire harness.

[0132] It should be noted that regression is a prediction action to form a value, which is different from the existing box detection method. The box detection method regresses to form a detection rectangular box, while the regression model of the present application regresses the adjacent position of the line beam, that is, the coordinate value of the pixel point corresponding to the line beam obstacle.

[0133] Exemplarily, the wire harness obstacle image is input into a trained classification model, and the image output by the model contains the wire harness adjacent area; the classification model has been trained with a large amount of labeled data and can identify the area containing the wire harness in the image.

[0134] Furthermore, the wire harness neighboring area identified by the classification model is input into the trained regression model, and the model outputs the precise position coordinates of the wire harness obstacle. The regression model can predict the precise position coordinates of the wire harness by learning the position features of the wire harness in the image.

[0135] For example, the prediction process of the regression model includes: for each pixel point in the adjacent area of ​​the wire beam, predicting the position of the wire beam closest to it, such as for pixel point A in the adjacent area of ​​the wire beam, drawing a perpendicular line to the wire beam, that is, finding the minimum distance between pixel point A and the line connecting the wire beam obstacle area, and then obtaining the position of pixel point B on the wire beam obstacle, then regressing through pixel point A to obtain position B, and so on, performing the above operation for each pixel point (A1, A2, ..., An) in the adjacent area of ​​the wire beam, and regressing to obtain the positions of all pixel points on the wire beam obstacle (B1, B2, ..., Bn), where B1, B2, ..., Bn are pixel points located on the wire beam obstacle, and then the regression model accurately predicts the overall position of the wire beam obstacle.

[0136] It can be understood that the regression model identifies the obstacles in the wire harness and accurately determines its specific location, solving the problem that the adjacent area of ​​the wire harness obtained by neighborhood segmentation does not accurately locate the wire harness.

[0137] In this way, the present application can optimize the performance of each model separately and improve the overall detection accuracy by decomposing the model prediction into two steps: classification and regression. The classification model first narrows the scope of the area that needs to be accurately positioned, reduces the computational burden of the regression model, and improves processing efficiency. The trained regression model can then be used to perform accurate regression positioning, fit the irregular shapes of the wiring harness obstacles, identify and accurately locate the wiring harness, and then output the specific position of the wiring harness.

[0138] It should be noted that the classification model is combined with the regression model to obtain the wire harness detection model, in which the classification model and the regression model are decoupled. Since the classification model itself has the risk of boundary instability, the classification model is only used to identify the wire harness adjacent area and does not accurately locate its position; instead, the regression model is used later to accurately locate the wire harness obstacles.

[0139] Optionally, the training process of the harness detection model includes:

[0140] Acquire a training data set; the training data set includes a plurality of training samples, each training sample includes a wire harness obstacle image and position coordinates of the wire harness obstacle in the wire harness obstacle image;

[0141] The training data set is input into the wire harness detection model for training to obtain a trained wire harness detection model;

[0142] The training process includes: determining the wire beam neighboring area of ​​each wire beam obstacle image, and determining the minimum distance corresponding to the line connecting the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and performing regression training using the line corresponding to the minimum distance, the position coordinates of the pixel points corresponding to each line, and the position coordinates of the wire beam obstacle.

[0143] For example, in the process of training the wire harness detection model, a large number of image data sets containing wire harness obstacles may be collected, and the image data sets include a plurality of training samples, each of which includes the image itself and the position coordinates of the wire harness obstacle.

[0144] Furthermore, the training data set is input into a wire harness detection model for training. The model may be a deep learning model, such as a convolutional neural network model, which is not specifically limited in the embodiments of the present application. During the training process, the model first identifies the wire harness neighborhood in each image, and then for each wire harness neighborhood, calculates the distance between the position coordinates of each pixel point and the position coordinates of the wire harness obstacle, and determines the line with the shortest distance among these lines to accurately locate the wire harness obstacle.

[0145] Furthermore, regression training is performed using the lines corresponding to the minimum distance, the position coordinates of the pixel points corresponding to each line, and the position coordinates of the wire harness obstacles, so that the model learns these relationships to predict the precise position of the wire harness obstacles in the new image.

[0146] In this way, by calculating the minimum distance and performing regression training, the model can accurately locate the position of the wiring harness obstacle and improve the detection accuracy. Furthermore, through precise regression training, the model can reduce the error in position prediction and improve the overall reliability. Among them, the wiring harness detection model can automatically learn the complex features in the image, reducing the dependence on manual work.

[0147] Optionally, the wire harness detection model includes a classification model and a regression model; the training data set is input into the wire harness detection model for training to obtain a trained wire harness detection model, including:

[0148] For each wire beam obstacle image, obtaining a wire beam neighboring area in the labeled wire beam obstacle image;

[0149] Inputting a plurality of wire beam obstacle images and a wire beam neighboring region corresponding to each wire beam obstacle image into a classification model for training to obtain a trained classification model;

[0150] Inputting the position coordinates of the plurality of wire harness adjacent regions and the wire harness obstacles in the wire harness adjacent regions into the regression model for training, thereby obtaining a trained regression model;

[0151] The trained classification model and the trained regression model are aggregated to obtain a trained wire harness detection model.

[0152] In the embodiment of the present application, the wire harness adjacent area is labeled for each wire harness obstacle image, which can be done manually to ensure that the wire harness adjacent area in each image is identified. It should be noted that the embodiment of the present application does not specifically limit the method for labeling the wire harness adjacent area in the wire harness obstacle image.

[0153] Exemplarily, multiple wire harness obstacle images and their corresponding wire harness adjacent areas are input into a classification model, and the classification model learns features in the images to identify which areas may contain wire harness obstacles, thereby obtaining a trained classification model that can effectively identify wire harness adjacent areas.

[0154] The identified multiple wire harness adjacent areas and their corresponding wire harness position coordinates are input into the regression model. The regression model learns the relationship between the pixel points in these areas and the wire harness position coordinates in order to accurately predict the position of the wire harness, thereby obtaining a trained regression model that can accurately locate the position of the wire harness obstacle in the identified wire harness adjacent areas.

[0155] Furthermore, the trained classification model and regression model are summarized to form a complete harness detection model, so that the harness detection model can identify the harness vicinity area, and then accurately locate the position of the harness obstacle in the harness vicinity area.

[0156] In this way, by combining the classification model and the regression model, the possible adjacent areas of the wiring harness can be identified, and then the position of the wiring harness can be accurately located to improve the overall detection accuracy. The independent training and optimization of the classification model and the regression model make the overall design more modular and easy to maintain and expand. Among them, the classification model first narrows the scope of the area that needs to be accurately processed, effectively filters out irrelevant areas, reduces the computational burden of the regression model, and improves processing efficiency. Then the regression model accurately locates the wiring harness obstacles and improves positioning accuracy.

[0157] Optionally, the training process of the regression model includes:

[0158] Obtain the position coordinates of each pixel point in the adjacent area of ​​the line beam;

[0159] For each pixel point, using the position coordinates of the pixel point and the position coordinates of the beam obstacle, multiple distances from the pixel point to multiple pixel points corresponding to the beam obstacle are calculated respectively;

[0160] Determine a minimum distance value among multiple distances, and determine a connection line between pixel points corresponding to the minimum distance value;

[0161] Regression training is performed by determining the connecting lines between the pixel points corresponding to the minimum distance, the position coordinates of the pixel points corresponding to each connecting line, and the position coordinates of the harness obstacles.

[0162] In the embodiment of the present application, the position coordinates of each pixel point are extracted from the adjacent area of ​​the wire beam to calculate the distance to the wire beam obstacle.

[0163] Exemplarily, for each pixel point, multiple distances between it and the position coordinates of the wiring harness obstacle are calculated. Further, among the multiple calculated distances, the minimum distance is found. This minimum distance represents the closest contact point between the pixel point and the wiring harness obstacle. Then, a line between the pixel point and the pixel point corresponding to the minimum distance is determined. The line is used to define the direct relationship between the pixel point and the wiring harness obstacle.

[0164] Furthermore, regression training is performed using these lines, the position coordinates of the pixel points corresponding to each line, and the position coordinates of the wire harness obstacles, so that the regression model can predict the precise position of the wire harness obstacles in new data by learning these relationships.

[0165] In this way, by utilizing the minimum distance between pixel points, the model can accurately capture the position of the harness obstacles and improve positioning accuracy. The minimum distance is determined by precise distance calculation and connection, so that the model can reduce errors in position prediction and improve overall reliability.

[0166] It is understandable that the above methods can help the model understand the spatial relationships and geometric features in the image, so that it can better adapt to different images and environments.

[0167] Optionally, avoiding obstacles in the wiring harness based on the position coordinates includes at least one of the following situations:

[0168] The cleaning route is replanned based on the position coordinates of the wiring harness obstacles, and the self-moving cleaning device performs the cleaning task according to the cleaning route to avoid the wiring harness obstacles;

[0169] The cleaning restricted area is determined based on the position coordinates of the harness obstacle. After the self-mobile cleaning equipment cleans around the cleaning restricted area, it continues to perform the cleaning task according to the originally planned path.

[0170] The position coordinates of the wire harness obstacle are sent to the user terminal for display, and the wire harness obstacle is avoided in response to the user's control operation.

[0171] In some embodiments, after the self-mobile cleaning device determines the position coordinates of the wire harness obstacle, a path planning algorithm, such as the A* algorithm or the Dijkstra algorithm, can be used to recalculate the cleaning route to ensure that the wire harness obstacle is avoided. The self-mobile cleaning device can perform tasks according to the newly planned cleaning route, thereby effectively avoiding the obstacle.

[0172] In other embodiments, after the self-mobile cleaning device determines the position coordinates of the wiring harness obstacle, based on these position coordinates, it determines the cleaning restricted area that the self-mobile cleaning device should not enter, and then the self-mobile cleaning device cleans around the cleaning restricted area. After completion, it continues to perform the task according to the originally planned path to ensure that the equipment can operate efficiently in a complex environment.

[0173] It is understandable that re-planning the path or defining restricted areas can ensure that the self-mobile cleaning equipment completes the cleaning task in the shortest time and improves the overall efficiency.

[0174] In some other embodiments, after the self-mobile cleaning device determines the position coordinates of the wiring harness obstacle, the position coordinates of the wiring harness obstacle can be sent to a user terminal device, such as a smart phone or a computer, so that the user can view the position of the wiring harness obstacle on the terminal and instruct the device to avoid these wiring harness obstacles by controlling the operation; wherein the self-mobile cleaning device can adjust the path according to the user's instructions to avoid the wiring harness obstacles.

[0175] It can be understood that by sending the location of the harness obstacle to the user terminal, the user can participate in the control of the device, which increases the flexibility and customization of the operation. This provision of real-time information and interactive functions can enhance the user's sense of control and satisfaction with the device.

[0176] In this way, after the self-mobile cleaning device determines the position coordinates of the wire harness obstacle, it can avoid the wire harness obstacle in a variety of ways to adapt to different environmental changes and user needs. While improving application flexibility, it can also reduce the risk of collision between the device and the wire harness obstacle, thereby protecting the equipment and other items in the environment.

[0177] Optionally, the present application also provides a self-moving cleaning device control method, which is applied to a user terminal, and the method includes:

[0178] receiving the position coordinates of the wire harness in the wire harness obstacle image determined during the cleaning task sent by the mobile cleaning device, and visually displaying the position coordinates;

[0179] Wherein, determining the position coordinates of the wire beam in the wire beam obstacle image includes:

[0180] After identifying the obstacle and determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle is obtained;

[0181] Determine a wire beam neighboring area of ​​the wire beam obstacle image, determine a connecting line between the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and determine the position coordinates of the wire beam obstacle based on the connecting line; the wire beam neighboring area at least includes an envelope area of ​​the wire beam obstacle; the envelope area is a closed area created around the boundary of the wire beam obstacle area.

[0182] In the present application, by visually displaying the location coordinates of the harness obstacles on the user terminal, the user can intuitively understand the environment around the device and potential harness obstacles, so that the user can better understand the working status and environment of the device.

[0183] The user terminal may also be referred to as terminal equipment, user equipment (UE), mobile station (MS), mobile terminal, terminal, etc. In practical applications, the user terminal is, for example, a desktop computer, a notebook, a personal digital assistant (PDA), a smart phone, a tablet computer, a vehicle-mounted device, a wearable device (such as a smart watch, a smart bracelet), a smart home device (such as a smart display device), etc., and the embodiments of the present application do not specifically limit this.

[0184] Optionally, users can view the locations of harness obstacles detected by the self-mobile cleaning device in real time, which improves the transparency of equipment operation and enables users to have a clearer understanding of the equipment's behavior and decision-making process. Users can then make informed decisions based on the visualized locations of harness obstacles, such as adjusting the cleaning route of the self-mobile cleaning device or setting new cleaning restricted areas to improve cleaning efficiency and safety.

[0185] In this way, by displaying the position of the harness obstacle on the user terminal, the user can take preventive measures to avoid the collision between the self-moving cleaning device and the harness obstacle, thereby protecting the self-moving cleaning device and the surrounding environment.

[0186] It should be noted that the process of determining the position coordinates of the harness obstacle by the mobile cleaning device can refer to the description of the above embodiment, which will not be repeated here.

[0187] For example, Figure 3 A schematic diagram of a scenario of interaction between a user terminal and a self-moving cleaning device provided in an embodiment of the present application, such as Figure 3 As shown, the scene includes a sweeping robot 100 and a user terminal 200. After the sweeping robot 100 determines the position coordinates of the wiring harness obstacle, the position coordinates of the wiring harness obstacle can be sent to the user terminal 200 so that the user terminal 200 displays the position of the wiring harness obstacle. There are many black dots in the figure, which are the position coordinates of the wiring harness in the determined wiring harness obstacle image. Optionally, in the visual display, the position coordinates of the wiring harness can be connected to generate the approximate shape of the wiring harness obstacle. The embodiment of the present application does not specifically limit this process.

[0188] In this way, by providing real-time visual content and interactive functions, the opportunities for users to participate in equipment operation are increased, user experience and satisfaction are improved, and users can dynamically adjust the cleaning strategy of the equipment according to the real-time displayed visual content, more accurately control the behavior of the equipment, and reduce the possibility of misoperation, thereby reducing the risk of damage to the self-mobile cleaning equipment.

[0189] In the aforementioned embodiments, the control method of the self-moving cleaning device provided by the embodiments of the present application is introduced. In order to realize the functions in the method provided by the embodiments of the present application, the self-moving cleaning device as the execution subject may include a hardware structure and / or a software module, and the above functions are realized in the form of a hardware structure, a software module, or a hardware structure plus a software module. Whether one of the above functions is executed in the form of a hardware structure, a software module, or a hardware structure plus a software module depends on the specific application and design constraints of the technical solution.

[0190] For example, Figure 4A schematic diagram of the structure of a self-moving cleaning device control device provided in an embodiment of the present application is applied to a self-moving cleaning device, such as Figure 4 As shown, the device 400 includes:

[0191] The identification module 401 is used to identify an obstacle and determine that the obstacle is a wire harness obstacle during the self-mobile cleaning device performing a cleaning task, and then obtain a wire harness obstacle image;

[0192] A determination module 402, for determining the position coordinates of the wire beam in the wire beam obstacle image, and avoiding the wire beam obstacle based on the position coordinates;

[0193] The determination module 402 is specifically used for:

[0194] Determine a wire beam neighboring area of ​​the wire beam obstacle image, determine a connecting line between the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and determine the position coordinates of the wire beam obstacle based on the connecting line; the wire beam neighboring area at least includes an envelope area of ​​the wire beam obstacle; the envelope area is a closed area created around the boundary of the wire beam obstacle area.

[0195] Optionally, the determining module 402 includes a first determining unit, where the first determining unit is configured to:

[0196] The position coordinates of the wire beam obstacle are determined based on the distance of the connecting line; wherein the distance of each connecting line is the minimum value of the distance between the position coordinates of the corresponding pixel point in the wire beam adjacent area and the connecting line of the position coordinates of the wire beam obstacle.

[0197] Optionally, the determination module 402 includes an obstacle avoidance unit, and the obstacle avoidance unit is used to:

[0198] The cleaning route is replanned based on the position coordinates of the wiring harness obstacles, and the self-moving cleaning device performs the cleaning task according to the cleaning route to avoid the wiring harness obstacles;

[0199] The cleaning restricted area is determined based on the position coordinates of the harness obstacle. After the self-mobile cleaning equipment cleans around the cleaning restricted area, it continues to perform the cleaning task according to the originally planned path.

[0200] The position coordinates of the wire harness obstacle are sent to the user terminal for display, and the wire harness obstacle is avoided in response to the user's control operation.

[0201] Optionally, the determining module 402 includes a second determining unit, where the second determining unit is configured to:

[0202] Obtain a predefined neighborhood window, and calculate the neighborhood features of each pixel point in the beam obstacle image within the predefined neighborhood window;

[0203] According to the neighborhood features, the pixel points in the beam obstacle image are segmented by threshold value to obtain the beam neighboring area and the non-beam neighboring area;

[0204] Among them, the self-moving cleaning device does not process areas that are not adjacent to the wiring harness.

[0205] Optionally, the first determining unit is specifically configured to:

[0206] The wire harness obstacle image is input into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the wire harness obstacle image.

[0207] Optionally, the wire harness detection model includes a classification model and a regression model; a prediction unit, specifically used for:

[0208] Input the wire harness obstacle image into the trained classification model to obtain the wire harness adjacent area in the wire harness obstacle image;

[0209] The adjacent area of ​​the wire harness is input into the trained regression model to obtain the position coordinates of the wire harness obstacles in the adjacent area of ​​the wire harness.

[0210] Optionally, the wire harness detection model is a neural network model based on deep learning.

[0211] Optionally, the apparatus 400 further includes a training module, and the training module is used to:

[0212] Acquire a training data set; the training data set includes a plurality of training samples, each training sample includes a wire harness obstacle image and position coordinates of the wire harness obstacle in the wire harness obstacle image;

[0213] The training data set is input into the wire harness detection model for training to obtain a trained wire harness detection model;

[0214] The training process includes: determining the wire beam neighboring area of ​​each wire beam obstacle image, and determining the minimum distance corresponding to the line connecting the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and performing regression training using the line corresponding to the minimum distance, the position coordinates of the pixel points corresponding to each line, and the position coordinates of the wire beam obstacle.

[0215] Optionally, the harness detection model includes a classification model and a regression model; a training module, specifically used for:

[0216] For each wire beam obstacle image, obtaining a wire beam neighboring area in the labeled wire beam obstacle image;

[0217] Inputting a plurality of wire beam obstacle images and a wire beam neighboring region corresponding to each wire beam obstacle image into a classification model for training to obtain a trained classification model;

[0218] Inputting the position coordinates of the plurality of wire harness adjacent regions and the wire harness obstacles in the wire harness adjacent regions into the regression model for training, thereby obtaining a trained regression model;

[0219] The trained classification model and the trained regression model are aggregated to obtain a trained wire harness detection model.

[0220] Optionally, the training module includes a training submodule, which is used for training the regression model. The training submodule is specifically used for:

[0221] Obtain the position coordinates of each pixel point in the adjacent area of ​​the line beam;

[0222] For each pixel point, using the position coordinates of the pixel point and the position coordinates of the beam obstacle, multiple distances from the pixel point to multiple pixel points corresponding to the beam obstacle are calculated respectively;

[0223] Determine a minimum distance value among multiple distances, and determine a connection line between pixel points corresponding to the minimum distance value;

[0224] Regression training is performed by determining the connecting lines between the pixel points corresponding to the minimum distance, the position coordinates of the pixel points corresponding to each connecting line, and the position coordinates of the harness obstacles.

[0225] It should be noted that the specific implementation principle and effects of the above-mentioned self-moving cleaning equipment control device can refer to the relevant descriptions and effects corresponding to the above-mentioned embodiments, and will not be elaborated here.

[0226] Optionally, the present application further provides a self-moving cleaning device control device, which is applied to a user terminal, and includes a receiving module, which is used to:

[0227] receiving the position coordinates of the wire harness in the wire harness obstacle image determined during the cleaning task sent by the mobile cleaning device, and visually displaying the position coordinates;

[0228] Wherein, determining the position coordinates of the wire beam in the wire beam obstacle image includes:

[0229] After identifying the obstacle and determining that the obstacle is a wire harness obstacle, an image of the wire harness obstacle is obtained;

[0230] Determine a wire beam neighboring area of ​​the wire beam obstacle image, determine a connecting line between the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and determine the position coordinates of the wire beam obstacle based on the connecting line; the wire beam neighboring area at least includes an envelope area of ​​the wire beam obstacle; the envelope area is a closed area created around the boundary of the wire beam obstacle area.

[0231] It can be understood that the specific implementation principle and effects of the above-mentioned self-moving cleaning equipment control device can refer to the relevant descriptions and effects corresponding to the above-mentioned relevant embodiments, and no further details will be given here.

[0232] The present application also provides a schematic diagram of the structure of a self-moving cleaning device. Figure 5 A schematic diagram of the structure of a self-moving cleaning device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the device may include: a processor 501 and a memory 502 that is communicatively connected to the processor; the memory 502 stores a computer program; the processor 501 executes the computer program stored in the memory 502, so that the processor 501 executes the method described in any of the above embodiments.

[0233] The memory 502 and the processor 501 may be connected via a bus 503 .

[0234] An embodiment of the present application further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the method described in any of the aforementioned embodiments of the present application.

[0235] An embodiment of the present application further provides a chip for executing instructions, wherein the chip is used to execute the method described in any of the aforementioned embodiments as executed by the self-moving cleaning device in any of the aforementioned embodiments of the present application.

[0236] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the method described in any of the aforementioned embodiments performed by the self-moving cleaning device in any of the aforementioned embodiments of the present application can be implemented.

[0237] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0238] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the solution of this embodiment.

[0239] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The above-mentioned module-composed unit can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0240] The above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application.

[0241] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0242] The memory may include high-speed random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk storage, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0243] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0244] The above storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The storage medium can be any available medium that can be accessed by a general or special computer.

[0245] An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor and the storage medium can also be present in the self-mobile cleaning device or the main control device as discrete components.

[0246] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0247] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0248] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0249] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0250] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.

Claims

1. A method for controlling a self-moving cleaning device, characterized in that: Applied to a self-propelled cleaning device, the method comprises: During the process of the self-moving cleaning device performing a cleaning task, after identifying an obstacle and determining that the obstacle is a wire harness obstacle, acquiring a wire harness obstacle image; determining position coordinates of the wire beam in the wire beam obstacle image, and avoiding the wire beam obstacle based on the position coordinates; Wherein, determining the position coordinates of the wire beam in the wire beam obstacle image includes: Determine a wire beam neighboring area of ​​the wire beam obstacle image, determine a line between the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and determine the position coordinates of the wire beam obstacle based on the line; the wire beam neighboring area at least includes an envelope area of ​​the wire beam obstacle; the envelope area is a closed area created around the boundary of the wire beam obstacle area.

2. The method according to claim 1, characterized in that The determining the position coordinates of the harness obstacle based on the connecting line comprises: The position coordinates of the beam obstacle are determined based on the distance of the connecting line; wherein the distance of each connecting line is the minimum value of the distance between the position coordinates of the corresponding pixel point in the beam adjacent area and the connecting line of the position coordinates of the beam obstacle.

3. The method according to claim 1, characterized in that The determining of the line beam adjacent area of ​​the line beam obstacle image comprises: Obtain a predefined neighborhood window, and calculate neighborhood features of each pixel point in the beam obstacle image within the predefined neighborhood window; According to the neighborhood features, threshold segmentation is performed on pixel points in the beam obstacle image to obtain a beam neighboring area and a non-beam neighboring area; Wherein, the self-moving cleaning device does not process the non-wiring harness adjacent area.

4. The method according to claim 1, characterized in that: The determining the position coordinates of the wire beam in the wire beam obstacle image comprises: The wire harness obstacle image is input into a trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the wire harness obstacle image.

5. The method according to claim 4, characterized in that The wire harness detection model includes a classification model and a regression model; the inputting the wire harness obstacle image into the trained wire harness detection model to predict the position coordinates of the wire harness obstacle in the wire harness obstacle image includes: Inputting the wire beam obstacle image into a trained classification model to obtain a wire beam adjacent area in the wire beam obstacle image; The wire harness neighborhood area is input into a trained regression model to obtain the position coordinates of the wire harness obstacles in the wire harness neighborhood area.

6. The method according to claim 4, characterized in that The wire harness detection model is a neural network model based on deep learning.

7. The method according to claim 4, characterized in that The training process of the wire harness detection model includes: Acquire a training data set; the training data set includes a plurality of training samples, each training sample includes a wire beam obstacle image and position coordinates of the wire beam obstacle in the wire beam obstacle image; Inputting the training data set into the wire harness detection model for training to obtain a trained wire harness detection model; The training process includes: determining a wire beam neighboring area of ​​each wire beam obstacle image, and determining a minimum distance value corresponding to a line connecting the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and performing regression training using a line corresponding to the minimum distance value, the position coordinates of the pixel points corresponding to each line, and the position coordinates of the wire beam obstacle.

8. The method according to claim 7, characterized in that The wire harness detection model includes a classification model and a regression model; the training data set is input into the wire harness detection model for training to obtain a trained wire harness detection model, including: For each wire beam obstacle image, obtaining a marked wire beam adjacent area in the wire beam obstacle image; Inputting the plurality of wire beam obstacle images and the wire beam neighboring area corresponding to each wire beam obstacle image into a classification model for training to obtain a trained classification model; Inputting the position coordinates of the plurality of wire harness adjacent areas and wire harness obstacles in the wire harness adjacent areas into a regression model for training, thereby obtaining a trained regression model; The trained classification model and the trained regression model are aggregated to obtain a trained wire harness detection model.

9. The method according to claim 8, characterized in that The training process of the regression model includes: Obtaining the position coordinates of each pixel point in the adjacent area of ​​the line beam; For each pixel point, using the position coordinates of the pixel point and the position coordinates of the beam obstacle, respectively calculate multiple distances from the pixel point to multiple pixel points corresponding to the beam obstacle; Determine a minimum distance value among the multiple distances, and determine a connection line between pixel points corresponding to the minimum distance value; Regression training is performed by determining the connecting line between the pixel points corresponding to the minimum distance value, the position coordinates of the pixel points corresponding to each connecting line, and the position coordinates of the wire harness obstacle.

10. The method according to claim 1, characterized in that The avoiding the harness obstacle based on the position coordinates includes at least one of the following situations: Replanning a cleaning route based on the position coordinates of the wire harness obstacle, and the self-moving cleaning device performs a cleaning task according to the cleaning route to avoid the wire harness obstacle; Determine a cleaning restricted area based on the position coordinates of the harness obstacle, and after the self-moving cleaning device cleans around the cleaning restricted area, continue to perform the cleaning task according to the originally planned path; The position coordinates of the wire harness obstacle are sent to a user terminal for display, and the wire harness obstacle is avoided in response to a user's control operation.

11. A method for controlling a self-moving cleaning device, characterized in that: Applied to a user terminal, the method comprises: receiving the position coordinates of the wire harness in the wire harness obstacle image determined during the execution of the cleaning task and sent by the self-mobile cleaning device, and visually displaying the position coordinates; Wherein, determining the position coordinates of the wire beam in the wire beam obstacle image includes: After identifying the obstacle and determining that the obstacle is a wire harness obstacle, acquiring a wire harness obstacle image; Determine a wire beam neighboring area of ​​the wire beam obstacle image, determine a line between the position coordinates of each pixel point in the wire beam neighboring area and the position coordinates of the wire beam obstacle, and determine the position coordinates of the wire beam obstacle based on the line; the wire beam neighboring area at least includes an envelope area of ​​the wire beam obstacle; the envelope area is a closed area created around the boundary of the wire beam obstacle area.

12. A self-moving cleaning device, characterized in that: The self-moving cleaning device includes: a processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 10.

Images