Obstacle recognition method, control system, robot, terminal and medium

Abstract

An obstacle recognition method, a control system, a robot, a terminal and a medium. The method comprises: acquiring obstacle information that is input by a user (S101); and inputting the obstacle information into a processing model in a cleaning robot, such that the processing model identifies, from an input image, an object matching the category of the obstacle information (S102).

Description

Obstacle recognition methods, control systems, robots, terminals, and media Cross-references to related applications

[0001] This application claims priority to Chinese patent application No. 202410917046.3, filed on July 9, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of automatic control, and more particularly to an obstacle recognition method, control system, robot, terminal, and medium. Background Technology

[0003] Nowadays, cleaning robots with automatic cleaning capabilities are favored by most consumers. Taking robotic vacuum cleaners as an example, they can automatically clean up garbage in the user's home environment, improve home cleanliness, and free users from daily household cleaning. However, since most users' home environments are relatively complex, there are many obstacles that affect the daily cleaning of robotic vacuum cleaners. These obstacles can cause collision noise at best, and at worst, cause the vacuum cleaner to get stuck and unable to continue cleaning. Summary of the Invention

[0004] This disclosure provides an obstacle recognition method, control system, robot, terminal, and medium through some embodiments, which can better identify obstacles in the environment.

[0005] In a first aspect of this disclosure, an obstacle recognition method is provided, the method comprising: acquiring obstacle information input by a user; and inputting the obstacle information into a processing model in a cleaning robot, such that the processing model identifies objects from the input image that conform to the category of the obstacle information.

[0006] In a second aspect of this disclosure, an obstacle recognition method is provided, the method comprising: acquiring obstacle information input by a user; and sending the obstacle information to a cleaning robot, such that the cleaning robot inputs the obstacle information into a preset processing model, and the processing model identifies objects from the input image that conform to the category of the obstacle information.

[0007] In a third aspect of this disclosure, a control system for a cleaning robot is provided, including a user terminal and a cleaning robot. The user terminal is used to acquire obstacle information input by a user and send the obstacle information to the cleaning robot. The cleaning robot is used to input the obstacle information into a preset processing model, such that the processing model identifies objects matching the category of the obstacle information from the input image.

[0008] In a fourth aspect of this disclosure, a cleaning robot is provided, including a processor and a memory, the memory storing a computer program executable on the processor, the computer program, when executed by the processor, performing the steps of the method as described in the first aspect.

[0009] In a fifth aspect of this disclosure, a user terminal is provided, including a processor and a memory, the memory storing a computer program executable on the processor, the computer program, when executed by the processor, implementing the steps of the method as described in the second aspect.

[0010] In a sixth aspect of this disclosure, a computer-readable storage medium is provided, including computer instructions stored thereon, which, when executed by a processor, implement the steps of the method described in the first or second aspect.

[0011] The above description is merely an overview of the technical solution provided in this disclosure. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other features and effects of this disclosure more obvious and understandable, the following are specific examples of the implementation methods of this disclosure. Attached Figure Description

[0012] Figure 1 shows a schematic diagram of the structure of a cleaning robot according to some embodiments of the present disclosure;

[0013] Figure 2 shows a flowchart of an obstacle recognition method according to some embodiments of the present disclosure;

[0014] Figure 3 shows a flowchart of image recognition steps according to some embodiments of the present disclosure;

[0015] Figure 4 illustrates a data processing schematic diagram of a processing model according to some embodiments of the present disclosure;

[0016] Figure 5 illustrates a data processing schematic diagram of a processing model according to other embodiments of this disclosure;

[0017] Figure 6 illustrates a flowchart of an obstacle recognition method according to other embodiments of the present disclosure; and

[0018] Figure 7 shows a schematic diagram of the structure of a control system according to some embodiments of the present disclosure. Detailed Implementation

[0019] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. It should be noted that the dimensions of the components may be exaggerated in the drawings for clarity of illustration. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0020] It should be noted that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The term "at least one" includes one or more cases, while the term "multiple" includes two or more cases. The terms "first," "second," etc., are used only for distinction and do not restrict the number or sequence of objects. The terms "before," "after," "above," "below," "left," "right," etc., are only used to indicate relative positional relationships. When the absolute position of the described objects changes, the relative positional relationship may also change accordingly.

[0021] A cleaning robot is an intelligent cleaning device with self-moving capabilities, used to clean target objects in the environment. For example, a cleaning robot can be a sweeping robot, a mopping robot, a combined sweeping and mopping robot, a floor polishing robot, or a lawnmower robot. The embodiments in this disclosure primarily use a sweeping robot as an example for illustration.

[0022] Taking a cleaning robot as an example, Figure 1 shows a schematic diagram of the structure of a cleaning robot according to some embodiments of the present disclosure. It should be noted that the structure and shape of the cleaning robot shown in Figure 1 are for illustrative purposes only and are not intended to limit the scope of the invention. As shown in Figure 1, the cleaning robot may include a main body 110, a sensing module 120, a controller, a drive module, cleaning components, a power module, and a human-machine interaction module 130. As shown in Figure 1, the main body 110 includes a front portion 111 and a rear portion 112, and has an approximately circular shape (both front and rear are circular). It may also have other shapes, including but not limited to an approximately D-shaped shape with a circular front and rear, and a rectangular or square shape with a circular front and rear.

[0023] For example, the sensing module 120 includes a position determination device 121 located on the machine body 110, a collision sensor located on the forward collision structure 122 of the forward part 111 of the machine body 110, a proximity sensor (wall sensor) located on the side of the machine, a cliff sensor located at the bottom of the machine body 110, and sensing devices such as magnetometer, accelerometer, gyroscope, and odometer located inside the machine body 110, which are used to provide the controller with various position information and motion status information of the machine.

[0024] As shown in Figure 1, the forward portion 111 of the main body 110 can carry the forward collision structure 122. During the cleaning process, when the drive wheel module propels the cleaning robot 10 to walk on the ground, the forward collision structure 122 detects one or more events in the travel path of the cleaning robot 10 through a sensor system installed on it, such as a collision sensor or a proximity sensor (such as an infrared sensor). The cleaning robot 10 can respond to the events detected by the forward collision structure 122, such as obstacles or walls, by controlling the drive module to make the cleaning robot 10 respond to the events, such as performing obstacle avoidance operations by moving away from obstacles.

[0025] For example, the controller may be located on a circuit board within the main body 110, including non-transitory memory (such as hard disk, flash memory, and random access memory) and processors (such as central processing unit and application processor).

[0026] For example, the drive module can manipulate the robot body 110 to travel across the ground based on drive commands with distance and angle information. For example, the cleaning components may include dry cleaning components and / or wet cleaning components; specific structures can be found in related technologies. When the cleaning robot 10 is in working mode, i.e., performing a cleaning task, it can clean target surfaces such as the ground using the cleaning components.

[0027] For example, the power module may include rechargeable batteries, such as nickel-metal hydride batteries and lithium batteries. The rechargeable batteries can be charged by connecting to electrodes on the cleaning base station via charging electrodes located on the side or bottom of the machine body 110.

[0028] For example, the human-machine interaction module 130 may include buttons on the main control panel for users to select functions. The human-machine interaction module 130 may also include one or more of a display screen, indicator lights, and speakers to show the user the current machine mode or function selection options. The human-machine interaction module 130 may also include a microphone to receive user voice commands to achieve voice control functionality. In some embodiments, the user may also interact with the cleaning robot 10 through a client installed on a user terminal that establishes a communication connection with the cleaning robot 10.

[0029] Of course, the structure of the cleaning robot 10 listed above is only exemplary, and the cleaning robot 10 may also include other structures, which are not limited in this disclosure.

[0030] The proper functioning of a cleaning robot depends on its ability to identify obstacles in its environment. For example, during its movement, the robot needs to avoid obstacles to prevent collisions or getting stuck.

[0031] Figure 2 shows a flowchart of an obstacle recognition method according to some embodiments of the present disclosure, which can be performed by a cleaning robot 10. As shown in Figure 2, the obstacle recognition method may include at least the following steps S101 and S102.

[0032] In step S101, obstacle information input by the user is obtained.

[0033] In step S102, obstacle information is input into the processing model in the cleaning robot so that the processing model can identify objects that match the category of obstacle information from the input image.

[0034] Most cleaning robots operate in complex environments with many unique obstacles that affect their daily cleaning work. By using the obstacle recognition method provided in some embodiments of this disclosure, users can input obstacle information into the cleaning robot according to the actual usage scenario and usage habits, and customize the types of obstacles that the cleaning robot can recognize. This helps the cleaning robot adapt to personalized usage environments, better recognize obstacles in the environment, and thus ensure the normal operation of the cleaning robot.

[0035] In some embodiments, the cleaning robot also pre-stores basic categories, which are general categories set at the factory. The types of basic categories are relatively limited and cannot cover all obstacle categories that need to be identified in all user scenarios. Some objects will still not be recognized, thus affecting subsequent obstacle avoidance and / or other processing. Based on the ability to identify obstacles of the basic categories, customizing the obstacle categories that the cleaning robot can recognize based on obstacle information input by the user helps to more comprehensively cover the obstacle categories that need to be identified in actual use scenarios.

[0036] In step S201, the obstacle information is information about obstacles that the user expects the cleaning robot to be able to recognize. In some embodiments, the obstacle information may include obstacle category information and / or additional information. Category information describes the category of the obstacle, and additional information describes the characteristics of the obstacle other than its category. When the obstacle information includes both category information and additional information, it is beneficial to describe the obstacles that the user expects the cleaning robot to be able to recognize in more detail, thereby helping the cleaning robot to more accurately identify obstacles that match the user's expectations. For example, if the obstacle that the user expects the cleaning robot to be able to recognize is a "dog," the obstacle information may include a description of this "dog," including not only the category information "dog" but also some additional information describing the characteristics of the dog, such as "a dog carrying a bone."

[0037] In some embodiments, where obstacle information includes additional information, the additional information may include, but is not limited to, one or more of the obstacle's shape, color, state, and spatial location.

[0038] For example, if a user describes the obstacle they want to avoid as "a round black lamp holder on the ground," then the obstacle information includes the obstacle's category: lamp holder, shape: round, color: black, and spatial location: on the ground. As another example, if a user describes the obstacle they want to avoid as "a rotten apple on the ground," then the obstacle information includes the obstacle's category: apple, condition: rotten, and spatial location: on the ground. Yet another example is if a user describes the obstacle they want to avoid as "a square black object against a wall," then the obstacle information includes the obstacle's shape: square, color: black, and spatial location: against a wall.

[0039] It should be noted that the additional information may also include other characteristics of the obstacle, which should be determined according to the needs of the actual application scenario, and this disclosure does not impose any restrictions on this. For example, if the environment in which the cleaning robot is used is divided into multiple areas, the additional information may also include the area where the user expects the obstacle to be identified, such as the living room or bedroom.

[0040] There are several ways to implement user input of obstacle information. For example, a user can interact with a client installed on a user terminal to input obstacle information, which is then sent to the cleaning robot by the user terminal. The user terminal and the cleaning robot are communicatively connected; for example, they can connect via wireless communication methods such as Bluetooth or Wi-Fi. The user terminal can be a mobile terminal such as a smartphone, tablet, or wearable device (such as a smartwatch or bracelet). The client installed on the user terminal can be used to control the cleaning robot and display its status to the user. Alternatively, the user can also input obstacle information by interacting with the cleaning robot.

[0041] There are various ways to input obstacle information, which can be set according to the actual needs of the product. For example, the input format of obstacle information can include one or more of text, voice, and images. For example, users can input obstacle information through a single method of text, voice, and images, or through a combination of text and voice, text and images, voice and images, or text, voice, and images.

[0042] In some embodiments, the process by which a cleaning robot acquires obstacle information input by a user may include: providing an obstacle information input channel to the user in response to an operation performed by the user to trigger obstacle information input; and acquiring the obstacle information input by the user through the input channel.

[0043] Users can interact with the user terminal to perform actions that trigger obstacle information input. Alternatively, users can interact with the cleaning robot to perform actions that trigger obstacle information input. Various actions can be used to trigger obstacle information input, such as button operations, voice activation, gesture operations, or touch operations (e.g., click, swipe, or tap), which can be configured according to the specific needs of the product.

[0044] There can be multiple types of input channels for obstacle information. The following explanation uses two exemplary input channels as examples.

[0045] The first exemplary input channel displays an input interface for obstacle information to the user and acquires the obstacle information input by the user through the input interface. In some embodiments, the input interface can be displayed to the user through a user terminal. In other embodiments, the input interface can also be displayed to the user through the screen of a cleaning robot.

[0046] In some embodiments, the process by which the cleaning robot acquires obstacle information input by a user through an input interface may include: acquiring obstacle information input by the user via voice and / or text on the input interface. For example, the input interface may have a text input box, from which the user can input a description of the obstacle they expect the cleaning robot to recognize. Another example is the input interface may have a voice input button, allowing the user to click the voice input button and then input a description of the obstacle they expect the cleaning robot to recognize via voice. Yet another example is the input interface may have both a text input box and a voice input button, allowing the user to choose between voice input, text input, or a combination of both.

[0047] In some embodiments, to facilitate user input, the input interface also includes input prompts for obstacle information, indicating which feature categories of the obstacles need to be input. For example, if the input interface includes multiple text input boxes, input prompts can be set next to each text input box. Exemplarily, the input interface may include category input boxes, shape input boxes, color input boxes, status input boxes, spatial location input boxes, and other descriptive input boxes to facilitate more accurate acquisition of obstacle information.

[0048] It should be noted that when the above input interface is displayed to the user via the user terminal, the obstacle information input by the user through voice and / or text on the input interface needs to be obtained through the user terminal first, and then the obtained obstacle information is sent to the cleaning robot.

[0049] In other embodiments, the process by which the cleaning robot acquires obstacle information input by the user through the input interface may include: acquiring an image input by the user, the image containing the obstacle to be described; and obtaining obstacle information based on the image. The input image can be a photo taken by the user; or it can be an image selected by the user from real-world images sent by the cleaning robot, eliminating the need for the user to take a photo. This way, when the user is unsure how to describe the obstacle they expect the cleaning robot to identify, they can simply input an image containing the obstacle to be described into the input interface, without needing to write the description themselves, thus reducing the difficulty of use for the user.

[0050] For example, when the above input interface is displayed to the user through a user terminal, the input interface may include an image upload option, allowing the user to select the image to be entered from locally stored images, or to input an image of the obstacle to be described in the environment in real time.

[0051] For example, a map of the cleaning robot's operating environment can be displayed to the user via a user terminal, marking obstacles identified by the robot. Clicking on these obstacle markers displays a real-world image of the obstacle along with its description. This means the cleaning robot identified the obstacle from this real-world image, which is generated based on images captured by the robot and includes the obstacle's bounding box. If the user believes the robot's obstacle identification is inaccurate, they can select this real-world image as input to obtain richer obstacle information, which can then be fed into the processing model to create a more detailed category for the obstacle, improving the accuracy of subsequent obstacle identification. For instance, if the user believes the robot's shoe identification is inaccurate, they can select a real-world image of the shoe to obtain richer descriptive terms, such as "red shoe."

[0052] In some embodiments, the process of obtaining obstacle information based on the image may include: sending the image to the cloud so that the cloud inputs the image into a preset object description model to obtain reference obstacle information; and obtaining obstacle information based on the reference obstacle information fed back by the cloud.

[0053] Object description models are pre-stored in the cloud. These models are trained on pre-built machine learning models, taking images as input and outputting descriptive information about the objects in the images. For example, the descriptive information may include, but is not limited to, one or more features such as the object's category, shape, color, state, and spatial location. By inputting an image containing the obstacle to be described into the object description model, the model outputs the obstacle's description information, which can be used as feedback for reference obstacles.

[0054] There are several ways to obtain obstacle information based on cloud-based reference obstacle information. In one optional implementation, the cloud-based reference obstacle information can be used as the actual obstacle information. This eliminates the need for further user input, reducing user intervention. It should be noted that in this case, the cloud can feed the obtained reference obstacle information back to the user terminal, which then sends this information as the obstacle information to the cleaning robot. Alternatively, the cloud can directly feed the obtained reference obstacle information back to the cleaning robot, which then inputs this information as user-input obstacle information into the processing model.

[0055] In another alternative implementation, reference obstacle information fed back from the cloud can be displayed to the user. This reference obstacle information serves as a reference for the user's input of obstacle information. At least a portion of the information selected by the user from the reference obstacle information can be acquired as obstacle information, or the obstacle information input by the user via voice and / or text at the input interface can be acquired. In other words, the reference obstacle information can be used to assist the user in inputting obstacle information, which helps improve the accuracy of the user's input and thus improves the accuracy of subsequent recognition.

[0056] For example, when displaying cloud-based reference obstacle information to the user via a user terminal, this reference obstacle information can be displayed on the input interface mentioned above. The user then inputs obstacle information based on this reference information. Method 1: The user can select some or all of the descriptive terms from the displayed reference obstacle information and fill them into the text input box on the input interface to complete the obstacle information input. Method 2: The user can refer to the descriptions in the reference obstacle information and input obstacle information separately via voice and / or text on the input interface. Methods 1 and 2 can be provided to the user, or both can be provided, allowing the user to choose their input method.

[0057] In other embodiments, the process of obtaining obstacle information based on the image may include: using a cleaning robot to identify the image and obtain obstacle information. For example, the object description model can be pre-stored in the cleaning robot. After the cleaning robot obtains the image input by the user at the input interface, it inputs the image into the object description model, and obtains the obstacle information based on the description information output by the object description model. For example, the description information output by the object description model can be used as the obstacle information. This eliminates the need for cloud access, which helps improve the efficiency of obstacle information input while reducing the difficulty of use for users.

[0058] The second exemplary input channel involves the cleaning robot providing voice prompts to the user, guiding them to input obstacle information via voice. The robot collects the user's voice signal and uses this signal to determine the obstacle information. In other words, the user can directly input obstacle information into the cleaning robot through voice interaction.

[0059] For example, after waking up the cleaning robot with a voice command, the user can issue a voice command to start inputting obstacle information. The cleaning robot responds to the voice command by outputting a voice prompt to the user, and then begins to collect the voice signal input by the user, thereby converting the voice signal into obstacle information.

[0060] For example, the wake-up word can be "Hi, ××", where "××" represents the name of the cleaning robot, which is determined according to the actual application scenario. The voice command can be "obstacle input" or "add obstacle", etc., and the voice prompt can be "please enter obstacle information". In practice, it can be set according to the needs of the application scenario, and this disclosure does not limit it.

[0061] After the cleaning robot obtains the obstacle information input by the user, it can perform the above step S102.

[0062] In some embodiments, the processing model in step S102 above is used to identify the types of obstacles that the cleaning robot needs to avoid during its movement. Accordingly, the obstacle information is related to the obstacles that the user expects the cleaning robot to avoid.

[0063] In other embodiments, the processing model can identify not only the categories of obstacles the cleaning robot needs to avoid during its movement, but also other categories of obstacles, to more comprehensively identify obstacles in the environment. In this case, in some embodiments, the cleaning robot can acquire not only the obstacle information input by the user, but also the corresponding usage label. For example, the usage label can be selected from two labels: obstacle avoidance and non-obstacle avoidance, set according to the needs of the actual application scenario. Based on this, after identifying target obstacles matching the category of the obstacle information from the real-world images collected by the cleaning robot through the processing model, the identified target obstacles can be processed based on the usage label corresponding to the obstacle information.

[0064] Purpose tags can be obtained before obstacle information is input into the recognition model. For example, when displaying an input interface for obstacle information to a user, the input interface can also include a list of purpose tags. This list can include multiple purpose tags, each representing the intended use of the cleaning robot after identifying that type of obstacle. In addition to inputting obstacle information, the user can also select the corresponding purpose tag from the tag list. It should be noted that one type of obstacle information can correspond to one or more purpose tags in the tag list, selected according to the needs of the actual application scenario.

[0065] Alternatively, the purpose label can be obtained after the cleaning robot identifies a target obstacle that matches the category of the obstacle information. For example, after identifying a target obstacle that matches the category of the obstacle information, the robot can then confirm the purpose of the target obstacle with the user, thereby obtaining the purpose label entered by the user.

[0066] In some embodiments, the cleaning robot pre-stores basic categories. These basic categories can be configured into the cleaning robot by relevant personnel before it leaves the factory. For example, they can include textual descriptions of various obstacle types that have been validated in a laboratory or textual descriptions of other obstacles with preset properties. Before the user inputs obstacle information, the cleaning robot can identify obstacles in these basic categories through a processing model. Since basic categories are difficult to cover all the personalized obstacle categories that need to be identified in all user scenarios, additionally customizing the obstacle categories that need to be identified based on the basic categories, through obstacle information input by the user during actual use, helps the cleaning robot to more accurately and comprehensively identify obstacles in the environment. Moreover, compared to simply having the user customize the obstacle categories, it helps to reduce user operations.

[0067] In some embodiments, to avoid consuming unnecessary computing resources, before inputting obstacle information into the processing model of the cleaning robot, the above method may further include: performing consistency matching between obstacle information and basic categories; if there is a category in the basic categories that matches the obstacle information, then outputting a prompt message to the user, the prompt message indicating that the category corresponding to the obstacle information already exists, and at this time the obstacle information input can be ignored, that is, step S102 is not required; if there is no category in the basic categories that matches the obstacle information, it means that the custom category corresponding to the obstacle information is a category not covered by the basic categories, then step S102 is executed.

[0068] For example, a cleaning robot can perform consistency matching between the acquired obstacle information and each pre-stored basic category. If the matching result reaches a preset threshold, the obstacle information is determined to match that basic category. For example, the matching result is expressed as a percentage, with a value range of 0 to 100%. The preset threshold can be 95%, 98%, or even 100%, depending on the needs of the actual application scenario. This disclosure does not impose any restrictions on this.

[0069] It should be noted that with the preset threshold set to 100%, a 100% matching rate is required, meaning the obstacle information must be completely consistent with the base category for the obstacle information to be considered a match. This allows for the addition of more granular custom categories under the base category, even when the base category covers a broad range, through user customization, which helps improve the accuracy of the recognition results.

[0070] After performing step S102, the cleaning robot can identify objects in the real-world image that match the category of obstacle information using the processing model described above.

[0071] In some embodiments, the obstacle recognition method described above further includes an image recognition step. As shown in FIG3, the image recognition step may include: step S201, inputting a real-scene image collected during the movement of the cleaning robot into a processing model; step S202, using the processing model to identify target obstacles belonging to a target category in the real-scene image and determine the location information of the target obstacles in the real-scene image, wherein the target category includes the category corresponding to the obstacle information input by the user.

[0072] The processing model in the embodiments of this disclosure is a pre-trained machine learning model. In some embodiments, as shown in FIG4, the input data of the processing model may include: obstacle information input by the user and real-world images collected by the cleaning robot, and the output data may include the position information of objects in the real-world images that match the category corresponding to the obstacle information. FIG4 takes the user's expected obstacle as the base of a table lamp as an example. Accordingly, the obstacle information is relevant information describing the base of the table lamp. The processing model can identify the base of the table lamp and determine its position in the real-world image (the position outlined by the box in the lower left corner of the real-world image in FIG4).

[0073] In other embodiments, the target category includes not only the category of obstacle information but also a pre-stored basic category in the cleaning robot. In this case, the input to the processing model also includes the pre-stored basic category, and the output data includes the position information of objects belonging to the basic category in the real-world image. In this scenario, the processing model can identify both target obstacles that match the category corresponding to the obstacle information and target obstacles belonging to the basic category. For ease of distinction, target obstacles identified from the real-world image that match the category corresponding to the obstacle information can be referred to as first target obstacles, and target obstacles belonging to the aforementioned basic category can be referred to as second target obstacles.

[0074] For example, the processing model may include a language neural network and an image neural network. The language and the image are processed by the language neural network and the image neural network respectively to extract features, and then the word embedding and image feature vector are fused together to infer the region in the image that is closer to the input language.

[0075] As shown in Figure 5, the processing model can include an encoder and a general object recognition model. First, the acquired obstacle information and pre-stored basic categories are input into the encoder and converted into vector representations. Then, these vectors, along with the acquired real-world image, are input into the general object recognition model. The model identifies a first detection box matching the category corresponding to the obstacle information and a second detection box matching the basic category from the real-world image, thereby outputting the location information of the first and second target obstacles in the real-world image. For example, the first detection box can be a bounding box containing the first target obstacle in the real-world image, and the second detection box can be a bounding box containing the second target obstacle in the real-world image.

[0076] It should be noted that when a user inputs obstacle information via voice, the speech can be converted into text by a pre-set speech recognition system, and the text can then be input into the encoder to be converted into a vector representation.

[0077] In some embodiments, where obstacle information includes the area where the obstacle is located (e.g., a living room or bedroom), the cleaning robot can record the area of ​​the captured real-world image. After the processing model identifies objects matching the category corresponding to the obstacle information from the real-world image, it can first compare the recorded area with the area included in the obstacle information. If they match, the object is determined to be a first target obstacle, and the location information of the first target obstacle is output. If they do not match, the object is determined not to be a first target obstacle, and the identified object is ignored. In other embodiments, in addition to inputting the captured real-world image into the processing model, the cleaning robot can also input the area of ​​the captured real-world image into the processing model. The processing model can combine the image features extracted from the real-world image with the area where the real-world image is located to identify objects matching the category corresponding to the obstacle information from the real-world image.

[0078] It is understood that after object recognition of the real-scene image through the above processing model, the obtained position information is the two-dimensional coordinates of the target obstacle in the real-scene image, i.e., image coordinates. To obtain the position of the target obstacle in the actual scene so that the movement path of the cleaning robot can be controlled based on the target obstacle's position, after determining the position information of the target obstacle in the real-scene image, the obstacle recognition method provided in some embodiments of this disclosure may further include: performing coordinate transformation on the position information to obtain the three-dimensional spatial coordinates of the target obstacle; and marking the target obstacle in a map constructed by the cleaning robot based on the three-dimensional spatial coordinates. It should be noted that when the above target category includes the category corresponding to the obstacle information input by the user and the pre-stored basic category, the target obstacle here includes a first target obstacle and a second target obstacle.

[0079] In some embodiments, the process of performing coordinate transformation on the position information to obtain the three-dimensional spatial coordinates of the target obstacle may include: acquiring depth information from a real-world image; and converting the two-dimensional coordinates of the target obstacle into three-dimensional spatial coordinates based on the depth information and pre-stored camera intrinsic parameters. The camera intrinsic parameters are the intrinsic parameters of the camera that acquired the real-world image. Depth information can be acquired using a depth sensor installed on the cleaning robot. For example, the depth sensor can be a laser distance sensor (LDS), a time-of-flight (TOF) module, or a structured light module, depending on the specific product.

[0080] For example, the two-dimensional coordinates (u, v) of the target obstacle, the pre-stored camera intrinsic parameters fx, fy, cx, cy, and the three-dimensional spatial coordinates (x, y, z) of the target obstacle satisfy the following system of equations: Where u and v are known, the depth information z of the corresponding point in the real scene image can be obtained through the depth sensor of the cleaning robot. That is, the above equation system can be simplified into a system of two linear equations in two variables, so as to find the coordinate values ​​x and y of the target obstacle in three-dimensional space, that is, to obtain the three-dimensional spatial coordinates of the target obstacle in the actual scene.

[0081] In some embodiments, where the target category includes the category corresponding to the obstacle information input by the user and a pre-stored basic category, the cleaning robot may label the first target obstacle and the second target obstacle differently in the constructed map. This allows the user to quickly distinguish between the two obstacles from the labels and promptly understand whether the cleaning robot has identified its customized obstacle category. Of course, in other embodiments, the labels for the first target obstacle and the second target obstacle may be the same, depending on the needs of the actual application scenario.

[0082] To improve the reliability of the cleaning robot's recognition of the first target obstacle, after performing step S202 above, which involves processing the model to identify the target obstacle belonging to the target category in the real-world image and determining the location information of the target obstacle in the real-world image, some embodiments of this disclosure provide an obstacle area recognition method that further includes a recognition result confirmation step. The recognition result confirmation step includes: if a target obstacle matching the category of obstacle information (hereinafter referred to as the first target obstacle) is identified from the real-world image, then confirming with the user whether the first target obstacle has been correctly identified.

[0083] It should be noted that the recognition result confirmation step can be performed after the first target obstacle is recognized and before the coordinate transformation and marking of the first target obstacle, or it can be performed after the coordinate transformation and marking of the first target obstacle. It can be set according to the needs of the actual application scenario, and this disclosure does not limit it.

[0084] In some embodiments, the process of confirming with the user whether the target obstacle has been correctly identified may include: displaying a real-world image and obstacle information to the user; and confirming the identification result based on the user's input to determine whether the target obstacle has been correctly identified. It should be noted that the real-world image displayed to the user here is the image from which the processing model has identified the first target obstacle. For example, the real-world image displayed to the user may also include the aforementioned first detection box, outlining the identified first target obstacle, to facilitate user confirmation.

[0085] For example, a recognition result confirmation interface can be displayed to the user through a user terminal. The recognition result confirmation interface contains a real-world image to be confirmed and obstacle information. The user can compare the real-world image with the obstacle information to determine whether the first target obstacle outlined in the real-world image is the obstacle described in the obstacle information. Then, the user can perform an operation to confirm the recognition result on the recognition result confirmation interface. In response to the user's operation to confirm the recognition result, the user terminal generates recognition result confirmation information and sends the recognition result confirmation information to the cleaning robot so that the cleaning robot can determine whether the target obstacle has been correctly identified based on the recognition result confirmation information.

[0086] For example, the recognition result confirmation interface can include a "Recognized Correctly" button and a "Recognized Incorrectly" button. Users can click the "Recognized Correctly" button if they believe the recognition is correct, and click the "Recognized Incorrectly" button if they believe it is incorrect. Alternatively, the recognition result confirmation interface can include a "Recognized Correctly" option, a "Recognized Incorrectly" option, and a "Confirm" button. Users can select either the "Recognized Correctly" or "Recognized Incorrectly" option and then click the "Confirm" button to confirm the recognition result.

[0087] In other embodiments, after the coordinate transformation and marking of the first target obstacle, the identification result confirmation step is performed. This identification result confirmation step may include: the cleaning robot sending a real-world image of the identified first target obstacle and the obstacle information corresponding to that image to a user terminal, so that the user terminal associates the received real-world image and obstacle information with the first target obstacle marked on the map; in response to the user clicking on the mark of the first target obstacle on the map, the robot displays the real-world image and obstacle information to the user; in response to the user's operation to confirm the identification result, the robot generates identification result confirmation information and sends it to the cleaning robot; the cleaning robot determines whether the target obstacle has been correctly identified based on the identification result confirmation information.

[0088] Of course, in other embodiments, after the cleaning robot identifies the first target obstacle, it can also confirm with the user via voice whether the first target obstacle has been correctly identified. For example, it can issue a voice prompt "××× identified, please confirm", where "×××" is determined based on the obstacle information actually entered by the user. After hearing the voice prompt, the user can input the recognition result confirmation information to the cleaning robot via voice.

[0089] If the first target obstacle is correctly identified, it means the cleaning robot has accurately identified the first target obstacle, and can proceed with further processing based on the identified location of the first target obstacle. In some embodiments, if the first target obstacle is incorrectly identified, the user is prompted to modify the obstacle information. In response to the user's operation to modify the obstacle information, the obstacle information in the input processing model is modified. For example, the cleaning robot can prompt the user to modify the obstacle information through a user terminal. The user terminal can respond to the user's operation to modify the obstacle information by sending a modification command to the cleaning robot to modify the obstacle information in the input processing model. Alternatively, the cleaning robot can also prompt the user to modify the obstacle information via voice.

[0090] For example, users can edit or delete obstacle information. For instance, when inputting text, they can add or delete descriptive words; when inputting images, they can take clearer pictures to obtain more accurate obstacle information.

[0091] In other embodiments, if the first target obstacle is incorrectly identified, the first target obstacle is ignored, and the identification continues in the next real-world image. Once the first target obstacle is identified again, the identification result confirmation step is performed again. If the first target obstacle has already been marked on the map after coordinate transformation and marking, and the identification result confirmation step is then performed, the mark on the first target obstacle needs to be removed from the map.

[0092] It should be noted that, to improve the reliability of second target obstacle identification, in other embodiments, if a target obstacle belonging to the aforementioned basic category (i.e., the second target obstacle) is identified from the real-world image, the user can be prompted to confirm whether the second target obstacle has been correctly identified. The confirmation process for the second target obstacle can be similar to that for the first target obstacle and can be configured according to the needs of the actual application scenario, which will not be detailed here. For example, if the user believes that the second target obstacle has been incorrectly identified, they can also input the real-world image corresponding to the second target obstacle into the input interface to obtain obstacle information with richer descriptive terms, which can then be input into the processing model to facilitate more accurate identification of the second target obstacle in the future.

[0093] After identifying the target obstacles (including the first target obstacle and the second target obstacle), the cleaning robot can be further controlled based on the position of the target obstacles. The specific control method can be determined according to the needs of the actual application scenario.

[0094] For example, in some application scenarios, the aforementioned target category refers to the category of obstacles that the cleaning robot needs to avoid. Therefore, after marking the target obstacles in the map constructed by the cleaning robot, the obstacle recognition method provided in some embodiments of this disclosure further includes: controlling the cleaning robot to avoid the target obstacles during movement based on the positions of the marked target obstacles in the map. This achieves obstacle avoidance for objects that the user expects the cleaning robot to avoid, enabling the cleaning robot to better adapt to personalized usage environments, reducing the risk of collisions or getting stuck due to some obstacles not being recognized, thus ensuring the normal operation of the cleaning robot and improving the user experience.

[0095] It should be noted that when the target category includes the category corresponding to the obstacle information input by the user and the pre-stored basic category, the target obstacle to be avoided here can include a first target obstacle and a second target obstacle. In other words, the cleaning robot can avoid obstacles not only for objects of the preset basic category, but also for objects that the user expects the cleaning robot to avoid.

[0096] In other application scenarios, the target obstacles marked on the map include not only obstacles that the cleaning robot needs to avoid, but also obstacles with other uses. In this case, the cleaning robot can be controlled to avoid the target obstacles during its movement based on the location of the target obstacles marked with their use labels on the map. For example, use labels can be pre-associated with each obstacle category that the processing model can recognize (including the category corresponding to the obstacle information input by the user and the pre-stored basic categories). When the cleaning robot marks target obstacles on the map, it can mark the location and use labels of the target obstacles for subsequent processing.

[0097] Figure 6 shows a flowchart of an obstacle recognition method according to other embodiments of the present disclosure, which can be executed on the user terminal described above. As shown in Figure 6, the obstacle recognition method may include at least the following steps S301 and S302.

[0098] In step S301, obstacle information input by the user is obtained.

[0099] In step S302, obstacle information is sent to the cleaning robot so that the cleaning robot inputs the obstacle information into a preset processing model, and the processing model identifies objects that match the category of obstacle information from the input image.

[0100] It should be noted that the specific implementation process and technical effects of steps S301 and S302 can be referred to the relevant descriptions of the method embodiments above, and will not be repeated here.

[0101] In some embodiments, the process of acquiring obstacle information input by the user may include: displaying an obstacle information input interface to the user in response to an operation performed by the user to trigger obstacle information input; and acquiring the obstacle information input by the user through the input interface. Specific implementation processes and technical effects can be found in the descriptions of the method embodiments above, and will not be repeated here.

[0102] In some embodiments, the process of acquiring obstacle information input by a user through an input interface may include: acquiring obstacle information input by the user through voice and / or text on the input interface. Specific implementation processes and technical effects can be found in the relevant descriptions of the method embodiments above, and will not be repeated here.

[0103] In some embodiments, obtaining obstacle information input by the user through an input interface includes: obtaining an image input by the user through the input interface, the image containing the obstacle to be described; and obtaining obstacle information based on the image. For example, the image may be a picture taken by the user; or, it may be an image selected by the user from real-world images sent by a cleaning robot. Specific implementation processes and technical effects can be referred to the relevant descriptions of the method embodiments above, and will not be repeated here.

[0104] In some embodiments, the process of obtaining obstacle information based on an image may include: sending the image to the cloud so that the cloud inputs the image into a preset category description model and outputs reference obstacle information; and obtaining obstacle information based on the reference obstacle information fed back by the cloud.

[0105] The process of acquiring obstacle information based on cloud-based reference obstacle information may include: using the cloud-based reference obstacle information as obstacle information; or, displaying the cloud-based reference obstacle information to the user, which serves as a reference for the user's input of obstacle information; acquiring at least a portion of the information selected by the user from the reference obstacle information as obstacle information; or, acquiring the obstacle information input by the user via voice and / or text at the input interface. Specific implementation processes and technical effects can be found in the descriptions of the method embodiments above, and will not be repeated here.

[0106] In other embodiments, obstacle information is obtained based on images, including sending the image to a cleaning robot so that the cleaning robot can identify the image and obtain obstacle information. Specific implementation processes and technical effects can be found in the descriptions of the method embodiments above, and will not be repeated here.

[0107] In some embodiments, the obstacle recognition method described above may further include: receiving a target real-scene image and obstacle information corresponding to the target real-scene image sent by a cleaning robot, wherein the target real-scene image is a real-scene image of a target obstacle that matches the category of the obstacle information; displaying the target real-scene image and obstacle information to a user; and generating recognition result confirmation information in response to an operation performed by the user to confirm the recognition result, and sending the recognition result confirmation information to the cleaning robot, wherein the recognition result confirmation information is used to determine whether the target obstacle has been correctly recognized. Specific implementation processes and technical effects can be referred to the relevant descriptions of the method embodiments above, and will not be repeated here.

[0108] In some embodiments, the obstacle recognition method may further include: if it is determined based on the recognition result confirmation information that the target obstacle is incorrectly recognized, prompting the user to modify the obstacle information; and in response to the user's operation to modify the obstacle information, sending a modification instruction to the cleaning robot to modify the obstacle information input to the processing model. Specific implementation processes and technical effects can be found in the descriptions of the method embodiments above, and will not be repeated here.

[0109] This disclosure also provides a control system for a cleaning robot in some embodiments. Figure 7 shows a schematic diagram of the control system according to some embodiments of this disclosure. As shown in Figure 7, the control system 1 includes a user terminal 20 and a cleaning robot 10. The user terminal 20 is communicatively connected to the cleaning robot 10, for example, via wireless communication methods such as Bluetooth or WIFI. The user terminal 20 has a client installed, which can be used to control the cleaning robot 10 and display the status of the cleaning robot 10 to the user.

[0110] User terminal 20 is used to acquire obstacle information input by the user and send the obstacle information to the cleaning robot; cleaning robot 10 is used to input the obstacle information into a preset processing model, so that the processing model can identify objects that match the category of obstacle information from the input image. Specific implementation processes and effects can be found in the relevant descriptions of the method embodiments above, and will not be repeated here.

[0111] Some embodiments of this disclosure also provide a cleaning robot, which includes a processor and a memory, the memory storing a computer program that can run on the processor. When the computer program is executed by the processor, it implements the steps of the obstacle recognition method performed by the cleaning robot in the above-described method embodiments, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0112] Some embodiments of this disclosure also provide a user terminal, which includes a processor and a memory, the memory storing a computer program executable on the processor. When executed by the processor, the computer program implements the steps of the obstacle recognition method executed by the user terminal in the above-described method embodiments, and achieves the same technical effect. To avoid repetition, it will not be described again here. Exemplarily, the user terminal can be a mobile terminal such as a mobile phone, tablet computer, or wearable device (such as a smartwatch or bracelet).

[0113] Some embodiments of this disclosure also provide a computer-readable storage medium, including computer instructions stored thereon. When executed by a processor, the computer instructions implement the steps of the obstacle recognition method provided in any of the above-described method embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0114] Some embodiments of this disclosure also provide a computer program product, including computer instructions. When the computer instructions are executed by a processor, they implement the steps of the obstacle recognition method provided in any of the above method embodiments and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0115] According to a first aspect of this disclosure, an obstacle recognition method is provided, comprising: acquiring obstacle information input by a user; and inputting the obstacle information into a processing model in a cleaning robot, such that the processing model identifies objects from the input image that conform to the category of the obstacle information.

[0116] In some implementations, acquiring obstacle information input by the user includes: providing the user with an obstacle information input channel in response to an operation performed by the user to trigger obstacle information input; and acquiring the obstacle information input by the user through the input channel.

[0117] In some implementations, providing the user with an input channel for obstacle information and obtaining the obstacle information input by the user through the input channel includes: displaying an input interface for the obstacle information to the user; and obtaining the obstacle information input by the user through the input interface.

[0118] In some implementations, obtaining obstacle information input by the user through the input interface includes: obtaining obstacle information input by the user through voice and / or text on the input interface.

[0119] In some implementations, obtaining obstacle information input by the user through the input interface includes: obtaining an image input by the user on the input interface, the image containing the obstacle to be described; and obtaining the obstacle information based on the image.

[0120] In some implementations, the image is a picture taken by the user; or, the image is a picture selected by the user from real-world images sent by the cleaning robot.

[0121] In some implementations, obtaining the obstacle information based on the image includes: sending the image to the cloud so that the cloud inputs the image into a preset object description model to obtain reference obstacle information; and acquiring the obstacle information based on the reference obstacle information fed back by the cloud.

[0122] In some implementations, obtaining the obstacle information based on the reference obstacle information fed back from the cloud includes: using the reference obstacle information fed back from the cloud as the obstacle information; or, displaying the reference obstacle information fed back from the cloud to the user, the reference obstacle information being used to provide a reference for the user to input the obstacle information, obtaining at least a portion of the information selected by the user from the reference obstacle information as the obstacle information, or obtaining the obstacle information input by the user through voice and / or text on the input interface.

[0123] In some implementations, obtaining the obstacle information based on the image includes: using the cleaning robot to identify the image to obtain the obstacle information.

[0124] In some implementations, providing the user with an input channel for obstacle information and obtaining the obstacle information input by the user through the input channel includes: outputting voice prompts to the user through the cleaning robot, the voice prompts being used to prompt the user to input obstacle information by voice; and collecting the voice signal input by the user and obtaining the obstacle information based on the voice signal.

[0125] In some implementations, the cleaning robot pre-stores basic categories. Before inputting the obstacle information into the processing model of the cleaning robot, the obstacle recognition method provided in the first aspect of this disclosure further includes: performing a consistency match between the obstacle information and the basic categories; if a category matching the obstacle information exists in the basic categories, outputting a prompt message to the user, the prompt message indicating that the category corresponding to the obstacle information already exists; and if no category matching the obstacle information exists in the basic categories, performing the step of inputting the obstacle information into the processing model of the cleaning robot.

[0126] In some implementations, after inputting the obstacle information into a processing model in the cleaning robot, the obstacle recognition method provided in the first aspect of this disclosure further includes: inputting real-scene images collected during the movement of the cleaning robot into the processing model; and identifying target obstacles belonging to a target category in the real-scene images and determining the position information of the target obstacles in the real-scene images through the processing model, wherein the target category includes the category corresponding to the obstacle information.

[0127] In some implementations, the target category also includes a base category pre-stored in the cleaning robot.

[0128] In some implementations, after determining the location information of the target obstacle in the real-world image, the obstacle recognition method provided in the first aspect of this disclosure further includes: performing coordinate transformation on the location information to obtain the three-dimensional spatial coordinates of the target obstacle; and marking the target obstacle in the map constructed by the cleaning robot based on the three-dimensional spatial coordinates.

[0129] In some implementations, after identifying target obstacles belonging to the target category in the real-world image through the processing model and determining the location information of the target obstacles in the real-world image, the obstacle identification method provided by the first aspect of this disclosure further includes: if a target obstacle matching the category of the obstacle information is identified from the real-world image, then confirming with the user whether the target obstacle has been correctly identified.

[0130] In some implementations, confirming with the user whether the target obstacle has been correctly identified includes: displaying the real-world image and obstacle information to the user; and determining whether the target obstacle has been correctly identified based on the user-inputted recognition result confirmation information.

[0131] In some implementations, after confirming with the user whether the target obstacle has been correctly identified, the obstacle identification method provided in the first aspect of this disclosure further includes: if the target obstacle is not correctly identified, prompting the user to modify the obstacle information; and modifying the obstacle information input to the processing model in response to the user's operation to modify the obstacle information; or, if the target obstacle has already been marked on the map, if the target obstacle is not correctly identified, canceling the mark, and when the target obstacle is identified again, performing the step of confirming with the user whether the target obstacle has been correctly identified again.

[0132] In some embodiments, after the target obstacle is marked in a map constructed by the cleaning robot, the obstacle recognition method provided in the first aspect of this disclosure further includes: controlling the cleaning robot to avoid the target obstacle during movement based on the position of the target obstacle marked in the map.

[0133] In some implementations, the obstacle information is input in one or more of the following formats: text, voice, and images.

[0134] In some implementations, the obstacle information includes obstacle category information and / or additional information, the additional information being used to describe characteristics of the obstacle other than its category.

[0135] In some implementations, the additional information includes at least one or more characteristics of the obstacle, such as its shape, color, state, and spatial location.

[0136] According to a second aspect of this disclosure, an obstacle recognition method is provided, the method comprising: acquiring obstacle information input by a user; and sending the obstacle information to a cleaning robot, such that the cleaning robot inputs the obstacle information into a preset processing model, and the processing model identifies objects from the input image that conform to the category of the obstacle information.

[0137] In some implementations, obtaining obstacle information input by a user includes: displaying an input interface for the obstacle information to the user in response to an operation performed by the user to trigger the input of the obstacle information; and obtaining the obstacle information input by the user through the input interface.

[0138] In some implementations, obtaining obstacle information input by the user through the input interface includes: obtaining obstacle information input by the user through voice and / or text on the input interface.

[0139] In some implementations, obtaining obstacle information input by the user through the input interface includes: obtaining an image input by the user on the input interface, the image containing the obstacle to be described; and obtaining the obstacle information based on the image.

[0140] In some implementations, the image is a picture taken by the user; or, the image is a picture selected by the user from real-world images sent by the cleaning robot.

[0141] In some implementations, obtaining the obstacle information based on the image includes: sending the image to the cloud so that the cloud inputs the image into a preset object description model to obtain reference obstacle information; and acquiring the obstacle information based on the reference obstacle information fed back by the cloud.

[0142] In some implementations, obtaining the obstacle information based on the reference obstacle information fed back from the cloud includes: using the reference obstacle information fed back from the cloud as the obstacle information; or, displaying the reference obstacle information fed back from the cloud to the user, the reference obstacle information being used to provide a reference for the user to input the obstacle information, obtaining at least a portion of the information selected by the user from the reference obstacle information as the obstacle information, or obtaining the obstacle information input by the user through voice and / or text on the input interface.

[0143] In some implementations, obtaining the obstacle information based on the image includes: sending the image to the cleaning robot so that the cleaning robot can identify the image and obtain the obstacle information.

[0144] In some implementations, the obstacle recognition method further includes: receiving a target real-scene image and obstacle information corresponding to the target real-scene image sent by the cleaning robot, wherein the target real-scene image is a real-scene image of a target obstacle that matches the category of the obstacle information; displaying the target real-scene image and the obstacle information to a user; and generating recognition result confirmation information in response to an operation performed by the user to confirm the recognition result, and sending the recognition result confirmation information to the cleaning robot, wherein the recognition result confirmation information is used to determine whether the target obstacle has been correctly recognized.

[0145] In some implementations, the obstacle recognition method provided in the second aspect of this disclosure further includes: if it is determined based on the recognition result confirmation information that the target obstacle is incorrectly recognized, prompting the user to modify the obstacle information; and in response to the user's operation to modify the obstacle information, sending a modification instruction to the cleaning robot to modify the obstacle information input to the processing model.

[0146] According to a third aspect of this disclosure, a control system for a cleaning robot is provided, including a user terminal and a cleaning robot. The user terminal is used to acquire obstacle information input by a user and send the obstacle information to the cleaning robot. The cleaning robot is used to input the obstacle information into a preset processing model, such that the processing model identifies objects matching the category of the obstacle information from the input image.

[0147] In a fourth aspect of this disclosure, a cleaning robot is provided, including a processor and a memory, the memory storing a computer program executable on the processor, the computer program, when executed by the processor, implementing the steps of the obstacle recognition method provided in the first aspect above.

[0148] According to a fifth aspect of this disclosure, a user terminal is provided, including a processor and a memory, the memory storing a computer program executable on the processor, the computer program, when executed by the processor, implementing the steps of the obstacle recognition method provided in the second aspect above.

[0149] According to a sixth aspect of this disclosure, a computer-readable storage medium is provided that stores computer instructions, which, when executed by a processor, implement the steps of the obstacle recognition method described in the first or second aspect.

[0150] Through the obstacle recognition method provided in some embodiments of this disclosure, users can input obstacle information into the cleaning robot and customize the types of obstacles that the cleaning robot can recognize. This helps the cleaning robot adapt to personalized usage environments and better recognize obstacles in the environment.

[0151] It should be noted that each embodiment in this disclosure focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to mutually. Where there is no conflict, features of the same embodiment and different embodiments of this disclosure can be combined with each other.

[0152] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this disclosure is limited to these examples; within the framework of this disclosure, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this disclosure as described above, which are not provided in detail for the sake of brevity.

[0153] Although exemplary embodiments of this disclosure have been described, those skilled in the art, upon learning the basic inventive concept, can make further changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the exemplary embodiments as well as all changes and modifications falling within the scope of this disclosure.

Claims

1. An obstacle recognition method, comprising: Obtain obstacle information input by the user; as well as The obstacle information is input into a processing model in the cleaning robot, so that the processing model can identify objects from the input image that match the category of the obstacle information.

2. The method according to claim 1, wherein, The process of obtaining obstacle information input by the user includes: In response to a user-executed operation to trigger obstacle information input, an obstacle information input channel is provided to the user; and Obstacle information input by the user through the input channel.

3. The method according to claim 2, wherein, Provide the user with an input channel for obstacle information, and obtain the obstacle information input by the user through the input channel, including: An input interface that displays the obstacle information to the user; and Obtain obstacle information input by the user through the input interface.

4. The method according to claim 3, wherein, Obtaining obstacle information input by the user through the input interface includes: Obtain obstacle information input by the user via voice and / or text on the input interface.

5. The method according to claim 3, wherein, Obtaining obstacle information input by the user through the input interface includes: Acquire an image input by the user on the input interface, the image containing the obstacle to be described; and Obstacle information is obtained based on the image.

6. The method according to claim 5, wherein, The image is a photo taken by the user; or, The image is selected by the user from the real-world images sent by the cleaning robot.

7. The method according to claim 5, wherein, Obtaining obstacle information based on the image includes: The image is sent to the cloud, so that the cloud can input the image into a preset object description model to obtain reference obstacle information; and Obtain the obstacle information based on the reference obstacle information fed back from the cloud.

8. The method according to claim 7, wherein, Based on the reference obstacle information fed back from the cloud, the obstacle information is obtained, including: The reference obstacle information fed back from the cloud is used as the obstacle information; or... The system displays the reference obstacle information fed back from the cloud to the user. The reference obstacle information is used to provide a reference for the user to input the obstacle information. The system obtains at least a portion of the information selected by the user from the reference obstacle information as the obstacle information, or obtains the obstacle information input by the user through voice and / or text on the input interface.

9. The method according to claim 5, wherein, Obtaining obstacle information based on the image includes: The cleaning robot identifies the obstacles in the image.

10. The method according to claim 2, wherein, Provide the user with an input channel for obstacle information, and obtain the obstacle information input by the user through the input channel, including: The cleaning robot outputs voice prompts to the user, which prompt the user to input obstacle information via voice; and Collect the user's voice input signal, and obtain the obstacle information based on the voice signal.

11. The method according to claim 1, wherein, The cleaning robot has pre-stored basic categories. Before inputting the obstacle information into the processing model in the cleaning robot, the method further includes: The obstacle information is matched consistently with the base category; If a category matching the obstacle information exists among the basic categories, a prompt message is output to the user, indicating that a category corresponding to the obstacle information already exists; and If no category matching the obstacle information exists in the basic categories, then the step of inputting the obstacle information into the processing model of the cleaning robot is performed.

12. The method according to any one of claims 1-11, further comprising, after inputting the obstacle information into the processing model in the cleaning robot: The real-world images captured during the movement of the cleaning robot are input into the processing model; as well as The processing model identifies target obstacles belonging to a target category in the real-world image and determines the location information of the target obstacles in the real-world image, wherein the target category includes the category corresponding to the obstacle information.

13. The method according to claim 12, wherein, The target category also includes the base category pre-stored in the cleaning robot.

14. The method according to claim 12, further comprising, after determining the position information of the target obstacle in the real-world image: The location information is transformed to obtain the three-dimensional spatial coordinates of the target obstacle; as well as Based on the three-dimensional spatial coordinates, the target obstacle is marked in the map constructed by the cleaning robot.

15. The method according to claim 14, further comprising, after identifying target obstacles belonging to the target category in the real-world image through the processing model and determining the location information of the target obstacles in the real-world image: If a target obstacle matching the category of the obstacle information is identified from the real-world image, the user is notified whether the target obstacle has been correctly identified.

16. The method according to claim 15, wherein, Confirming with the user whether the target obstacle has been correctly identified includes: Show the user the real-world image and the obstacle information; and Based on the recognition result confirmation information input by the user, it is determined whether the target obstacle has been correctly identified.

17. The method of claim 15, further comprising, after confirming with the user whether the target obstacle has been correctly identified: If the target obstacle is incorrectly identified, the user is prompted to modify the obstacle information. In response to the user's action to modify the obstacle information, the obstacle information input into the processing model is modified; or... If the target obstacle has already been marked on the map, and the target obstacle is not correctly identified, the mark is removed. When the target obstacle is identified again, the step of confirming with the user whether the target obstacle has been correctly identified is repeated.

18. The method of claim 14, further comprising, after marking the target obstacle in the map constructed by the cleaning robot: Based on the location of the target obstacles marked on the map, the cleaning robot is controlled to avoid the target obstacles during its movement.

19. The method according to claim 1, wherein, The obstacle information input format includes one or more of the following formats: Text, voice, and images.

20. The method according to claim 1, wherein, The obstacle information includes obstacle category information and / or additional information, the additional information being used to describe the characteristics of the obstacle other than its category.

21. The method according to claim 20, wherein, The additional information includes at least one or more features of the obstacle, such as its shape, color, state, and spatial location.

22. An obstacle recognition method, comprising: Obtain obstacle information input by the user; as well as The obstacle information is sent to the cleaning robot, which then inputs the obstacle information into a preset processing model, enabling the processing model to identify objects that match the category of the obstacle information from the input image.

23. The method according to claim 22, wherein, Obtain obstacle information input by the user, including: In response to a user-executed action to trigger obstacle information input, an input interface for the obstacle information is displayed to the user; and Obtain obstacle information input by the user through the input interface.

24. The method according to claim 23, wherein, Obtaining obstacle information input by the user through the input interface includes: Obtain obstacle information input by the user via voice and / or text on the input interface.

25. The method according to claim 23, wherein, Obtaining obstacle information input by the user through the input interface includes: Acquire an image input by the user on the input interface, the image containing the obstacle to be described; and Obstacle information is obtained based on the image.

26. The method of claim 25, wherein, The image is a photo taken by the user; or, The image is selected by the user from the real-world images sent by the cleaning robot.

27. The method according to claim 25, wherein, Obtaining obstacle information based on the image includes: The image is sent to the cloud, so that the cloud can input the image into a preset object description model to obtain reference obstacle information; and Obtain the obstacle information based on the reference obstacle information fed back from the cloud.

28. The method according to claim 27, wherein, Based on the reference obstacle information fed back from the cloud, the obstacle information is obtained, including: The reference obstacle information fed back from the cloud is used as the obstacle information; or... The system displays the reference obstacle information fed back from the cloud to the user. The reference obstacle information is used to provide a reference for the user to input the obstacle information. The system obtains at least a portion of the information selected by the user from the reference obstacle information as the obstacle information, or obtains the obstacle information input by the user through voice and / or text on the input interface.

29. The method according to claim 25, wherein, Obtaining obstacle information based on the image includes: The image is sent to the cleaning robot so that the cleaning robot can identify the image and obtain the obstacle information.

30. The method according to any one of claims 22-29, further comprising: The system receives a target real-world image and obstacle information corresponding to the target real-world image sent by the cleaning robot. The target real-world image is a real-world image of a target obstacle that matches the category of the obstacle information. Display the target real-world image and the obstacle information to the user; as well as In response to a user's action to confirm the recognition result, a recognition result confirmation message is generated and sent to the cleaning robot. The recognition result confirmation message is used to determine whether the target obstacle has been correctly identified.

31. The method of claim 30, further comprising: If the recognition result confirms that the target obstacle is incorrectly identified, the user is prompted to modify the obstacle information. as well as In response to a user-executed operation to modify the obstacle information, a modification instruction is sent to the cleaning robot to modify the obstacle information input into the processing model.

32. A control system for a cleaning robot, comprising a user terminal and a cleaning robot, wherein, The user terminal is used to acquire obstacle information input by the user and send the obstacle information to the cleaning robot; as well as The cleaning robot is used to input the obstacle information into a preset processing model, so that the processing model can identify objects that match the category of the obstacle information from the input image.

33. A cleaning robot, comprising a processor and a memory, the memory storing a computer program executable on the processor, the computer program, when executed by the processor, implementing the steps of the method as claimed in any one of claims 1-21.

34. A user terminal, comprising a processor and a memory, the memory storing a computer program executable on the processor, the computer program, when executed by the processor, implementing the steps of the method as described in any one of claims 22-31.

35. A computer-readable storage medium comprising computer instructions stored thereon, which, when executed by a processor, implement the steps of the method according to any one of claims 1-31.

Images