Self-moving device control method and apparatus, device, medium, and program product

Abstract

A self-moving device control method and apparatus, an electronic device, a computer readable storage medium, and a computer program product, which relate to the technical field of intelligent robots, and can be applied to a scenario where movement control is performed on a cleaning robot. The method comprises: acquiring image information and laser point cloud information of a surrounding environment; performing object recognition processing and result fusion processing on the basis of the image information and the laser point cloud information to obtain object types and position information of objects to be recognized in the surrounding environment; and determining an obstacle avoidance strategy of a self-moving device on the basis of the object types and the position information, the obstacle avoidance strategy being used for controlling to generate a moving path of the self-moving device. In the present application, image information and laser point cloud information of a surrounding environment are fused, and a more accurate object type classification result can be obtained on the basis of the fusion result, so that a more reasonable moving path can be generated for a self-moving device.

Description

Method, device, medium and program product for controlling mobile equipment CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to the application number 202410316162.X filed with the State Intellectual Property Office of China on March 19, 2024, entitled “Control Methods and Devices, Equipment, Media and Program Products for Self-Mobility Equipment,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] The present disclosure relates to the field of intelligent robot technology, and in particular to a control method for a self-moving device, a control device for a self-moving device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0003] With the development of artificial intelligence (AI) technology, a wide variety of intelligent robots have emerged, including industrial robots, primary intelligent robots, intelligent agricultural robots, intelligent home care robots, and advanced intelligent robots. For example, intelligent home care robots include sweeping robots, mopping robots, vacuum cleaners, and lawn mowers. These cleaning robots can automatically identify and avoid obstacles while working.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0005] The present disclosure aims to provide a method for controlling a mobile device, a device for controlling a mobile device, an electronic device, and a computer-readable storage medium.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0007] According to a first aspect of the present disclosure, a method for controlling a self-moving device is provided, comprising: acquiring image information and laser point cloud information of a surrounding environment; performing object recognition processing and result fusion processing based on the image information and the laser point cloud information to obtain item type and location information of an item to be identified in the surrounding environment; and determining an obstacle avoidance strategy for the self-moving device based on the item type and the location information, wherein the obstacle avoidance strategy is used to control and generate a moving path for the self-moving device.

[0008] In an exemplary embodiment of the present disclosure, the performing of object recognition processing and result fusion processing based on the image information and the laser point cloud information to obtain the item type and position information of the item to be recognized in the surrounding environment includes: performing object recognition processing based on the image information and the laser point cloud information to obtain an item type recognition result, the item type recognition result including an image recognition result and / or a point cloud recognition result; obtaining an image point cloud relationship between the image information and the laser point cloud information, the image point cloud relationship including a correspondence between pixel positions of the environmental image and ranging points of point cloud data; obtaining an item category point cloud and / or an item category image corresponding to the item type recognition result based on the image point cloud relationship; and performing the result fusion processing on the item type recognition result and the corresponding item category point cloud and / or the item category image to obtain the item type and the position information.

[0009] In an exemplary embodiment of the present disclosure, the performing of object recognition processing based on the image information and / or the laser point cloud information to obtain an object type recognition result includes: performing object recognition processing based on the image information to obtain an image recognition result of the object to be identified, the image recognition result including an object bounding box or segmented pixel information; and / or performing object recognition processing based on the laser point cloud information to obtain a point cloud recognition result of the object to be identified, the point cloud recognition result including an object category point.

[0010] In an exemplary embodiment of the present disclosure, the image recognition result includes an item bounding box or segmented pixel information, and the item type recognition result is fused with the corresponding item category point cloud and / or the item category image to obtain the item type and the location information, including: projecting the item bounding box or the segmented pixel information onto the item category point cloud to obtain point cloud features of the item category point cloud; classifying, filtering and identifying the image recognition result according to the point cloud features to obtain the item type, and determining the location information of the item type.

[0011] In an exemplary embodiment of the present disclosure, the image recognition result is classified, filtered and identified according to the point cloud features to obtain the item type, including: when the image recognition result is the first obstacle and the point cloud feature does not have height information, determining the item type as no obstacle; when the image recognition result is the first obstacle and the point cloud feature has height information, acquiring spatial attribute features of point cloud data based on the point cloud features; and determining the item type according to the spatial attribute features.

[0012] In an exemplary embodiment of the present disclosure, determining the item type based on the spatial attribute characteristics includes: when the spatial attribute characteristics are that the space is not closed, determining the item type as the first obstacle; when the spatial attribute characteristics are that the space is closed, determining the item type as the second obstacle.

[0013] In an exemplary embodiment of the present disclosure, the image recognition result is classified, filtered and identified according to the point cloud features to obtain the object type, including: when the image recognition result is a third obstacle and the point cloud feature has height information, determining the object type as no obstacle; when the image recognition result is the third obstacle and the point cloud feature does not have height information, obtaining reflection intensity information based on the point cloud feature; and determining the object type according to the reflection intensity information.

[0014] In an exemplary embodiment of the present disclosure, the fusing of the item type recognition result with the corresponding item category point cloud and / or the item category image to obtain the item type and the location information includes: based on the image point cloud relationship, performing position alignment processing on the point cloud recognition result and the item category image to obtain an initial fusion result, wherein the image pixels in the initial fusion result have ranging data; obtaining color features corresponding to the item category image; and filtering and identifying the initial fusion result based on the color features to obtain the item type and the location information.

[0015] In an exemplary embodiment of the present disclosure, the point cloud recognition result and the item category image are aligned based on the image-point cloud relationship to obtain an initial fusion result, including: projecting the item category points in the point cloud recognition result to the item category image based on the image-point cloud relationship to obtain the initial fusion result; or projecting the item category image to the point cloud recognition result based on the image-point cloud relationship to obtain the initial fusion result.

[0016] In an exemplary embodiment of the present disclosure, the object identification processing and result fusion processing based on the image information and the laser point cloud information to obtain the object type and position information of the object to be identified in the surrounding environment include: performing object identification processing based on the image information and the laser point cloud information to obtain image recognition results and point cloud recognition results; performing result fusion processing on the image recognition results and the point cloud recognition results to obtain the object type; obtaining the image-point cloud relationship, and determining the position information corresponding to the object type according to the image-point cloud relationship.

[0017] According to a second aspect of the present disclosure, a control device for a self-moving device is provided, comprising: an environmental information acquisition module for acquiring image information and laser point cloud information of a surrounding environment; an object type determination module for performing object recognition processing and result fusion processing based on the image information and the laser point cloud information, to obtain the object type and position information of the object to be identified in the surrounding environment; and a device control module for determining an obstacle avoidance strategy for the self-moving device based on the object type and the position information, wherein the obstacle avoidance strategy is used to control the generation of a moving path of the self-moving device.

[0018] In an exemplary embodiment of the present disclosure, the item type determination module includes a first type determination unit, which is used to: perform item recognition processing based on the image information and the laser point cloud information to obtain an item type recognition result, wherein the item type recognition result includes an image recognition result and / or a point cloud recognition result; obtain an image point cloud relationship between the image information and the laser point cloud information, wherein the image point cloud relationship includes a correspondence between pixel positions of an environmental image and ranging points of point cloud data; based on the image point cloud relationship, obtain an item category point cloud and / or an item category image corresponding to the item type recognition result; and perform the result fusion processing on the item type recognition result and the corresponding item category point cloud and / or the item category image to obtain the item type and the position information.

[0019] In an exemplary embodiment of the present disclosure, the first type determination unit includes a type recognition subunit, which is used to: perform object recognition processing based on the image information to obtain an image recognition result of the object to be identified, and the image recognition result includes an object bounding box or segmented pixel information; and / or perform object recognition processing based on the laser point cloud information to obtain a point cloud recognition result of the object to be identified, and the point cloud recognition result includes an object category point.

[0020] In an exemplary embodiment of the present disclosure, the image recognition result includes an item bounding box or segmented pixel information, and the first type determination unit includes a first type determination subunit, which is used to: project the item bounding box or the segmented pixel information onto the item category point cloud to obtain point cloud features of the item category point cloud; classify, filter and identify the image recognition result according to the point cloud features to obtain the item type, and determine the location information of the item type.

[0021] In an exemplary embodiment of the present disclosure, the first type determination subunit includes a first filtering subunit, which is used to: when the image recognition result is the first obstacle and the point cloud feature does not have height information, determine the object type as no obstacle; when the image recognition result is the first obstacle and the point cloud feature has height information, obtain spatial attribute features of the point cloud data based on the point cloud feature; and determine the object type according to the spatial attribute features.

[0022] In an exemplary embodiment of the present disclosure, the first type determination subunit includes a second filtering subunit, which is used to: when the spatial attribute feature is that the space is not closed, determine the item type as the first obstacle; when the spatial attribute feature is that the space is closed, determine the item type as the second obstacle.

[0023] In an exemplary embodiment of the present disclosure, the first type determination subunit includes a third filtering subunit, which is used to: when the image recognition result is a third obstacle and the point cloud feature has height information, determine the object type as no obstacle; when the image recognition result is a third obstacle and the point cloud feature does not have height information, obtain reflection intensity information based on the point cloud feature; and determine the object type according to the reflection intensity information.

[0024] In an exemplary embodiment of the present disclosure, the item type determination module includes a second type determination unit, which is used to: based on the image point cloud relationship, align the point cloud recognition result with the item category image to obtain an initial fusion result, where the image pixels in the initial fusion result have ranging data; obtain color features corresponding to the item category image; and filter and identify the initial fusion result based on the color features to obtain the item type and the location information.

[0025] In an exemplary embodiment of the present disclosure, the second type determination unit includes a fusion processing subunit, which is used to: project the item category points in the point cloud recognition result to the item category image based on the image point cloud relationship to obtain the initial fusion result; or project the item category image to the point cloud recognition result based on the image point cloud relationship to obtain the initial fusion result.

[0026] In an exemplary embodiment of the present disclosure, the item type determination module includes a third type determination unit, which is used to: perform item recognition processing based on the image information and the laser point cloud information to obtain image recognition results and point cloud recognition results; merge the image recognition results and the point cloud recognition results to obtain the item type; obtain the image-point cloud relationship, and determine the position information corresponding to the item type based on the image-point cloud relationship.

[0027] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the control method of the self-mobile device according to any one of the above items is implemented.

[0028] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the control method of the self-moving device according to any one of the above items is implemented.

[0029] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements any one of the above-described methods for controlling a self-mobile device.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0032] FIG1 schematically shows a flow chart of a method for controlling a self-moving device according to an exemplary embodiment of the present disclosure;

[0033] FIG2 shows a schematic diagram of identifying wire articles according to an exemplary embodiment of the present disclosure;

[0034] FIG3 shows a schematic diagram of identifying low obstacles according to an exemplary embodiment of the present disclosure;

[0035] FIG4 shows a schematic diagram of identifying stain information according to an exemplary embodiment of the present disclosure;

[0036] FIG5 schematically shows a block diagram of a control device for a self-moving device according to an exemplary embodiment of the present disclosure;

[0037] FIG6 schematically shows a block diagram of an electronic device according to an exemplary embodiment of the present disclosure;

[0038] FIG7 schematically shows a diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.

[0040] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, well-known structures, methods, devices, implementations, materials or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0041] The blocks shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. Specifically, these functional entities may be implemented in software, or in one or more software-hardened modules, or in different networks and / or processor devices and / or microcontroller devices.

[0042] When using a self-cleaning robot for floor cleaning, the robot can detect the distance between itself and obstacles to automatically avoid them. For example, when using a monocular image to identify obstacles, the red, green, and blue (RGB) color information can be transmitted to a neural network, and the pixel area of ​​the corresponding object position in the image is output, including a bounding box or a specific pixel area. However, when the robot is actually used, the information actually used is the point cloud data that projects the image pixels into the three-dimensional world. Generally speaking, the accuracy of the feature point cloud obtained based on image conversion is relatively poor, and one advantage of the time of flight (TOF) ranging method is that the ranging result is more accurate.

[0043] Based on this, in this example embodiment, a method for controlling a self-moving device is first provided. The method described in this disclosure can be implemented using a terminal device, wherein the terminal described in this disclosure can include a personal digital assistant (PDA), a portable media player (PMP), a navigation device, a pedometer, an intelligent robot, a sweeping robot, a mopping robot, and other mobile terminals. Figure 1 schematically shows a schematic diagram of the control method flow of a self-moving device according to some embodiments of the present disclosure. Referring to Figure 1, the control method of the self-moving device may include the following steps:

[0044] Step S110: Acquire image information and laser point cloud information of the surrounding environment.

[0045] According to some exemplary embodiments of the present disclosure, the surrounding environment may be the environment in which a mobile device is located during movement. Image information may be an image obtained by capturing images of the surrounding environment. Laser point cloud information may be information contained in a dataset of spatial points obtained by scanning the surrounding environment using a three-dimensional laser radar device, where each point cloud includes three-dimensional coordinates (XYZ) and laser reflection intensity (Intensity).

[0046] This embodiment uses a robot vacuum as an example. The self-moving device is a multi-sensor system that can include a laser radar (LiDAR) and a visual camera. As the self-moving device moves, the visual camera captures images of the surrounding environment, generating corresponding image information based on the captured images. The laser radar device can also continuously scan the surrounding environment, obtaining distance information for all points and lines in the environment. After imaging processing, point cloud data is generated, and the collected distance information for all points and lines is used as laser point cloud information.

[0047] Step S120 , performing object recognition processing and result fusion processing based on the image information and the laser point cloud information to obtain the object type and location information of the object to be recognized in the surrounding environment.

[0048] According to some exemplary embodiments of the present disclosure, object identification processing may be a process for identifying an object type based on at least one of image information and laser point cloud information. Result fusion processing may be a process for fusing the identification results obtained from performing object identification processing on image information and laser point cloud information to determine the object type. The object to be identified may be an object in the surrounding environment. The object type may be the specific type of an object in the surrounding environment. Location information may be information indicating the location of an object in the surrounding environment.

[0049] After obtaining image information and laser point cloud information of the surrounding environment, object recognition processing can be performed based on the image information and laser point cloud information respectively. During the object recognition process, object recognition processing can be performed based solely on image information, solely on laser point cloud information, or a combination of the two. If object recognition processing is performed using image information and laser point cloud information separately, the recognition results obtained from the two are fused, such as performing an intersection operation on the two recognition results, to obtain the item type and location information of the object to be identified in the surrounding environment.

[0050] If only image information is used for object recognition, after obtaining the image recognition results, the image recognition results can be fused with the laser point cloud information to determine the item type and location information of the item to be identified in the surrounding environment. If only laser point cloud information is used for object recognition, after obtaining the point cloud recognition results, the point cloud recognition results can be fused with the image information to determine the item type and location information of the item to be identified in the surrounding environment. Because the image information and laser point cloud information are combined in the process of determining the item type, not only the image recognition results can be obtained, but also the point cloud distance information can be obtained, further improving the accuracy of the recognition results.

[0051] In step S130 , an obstacle avoidance strategy for the mobile device is determined based on the type and location information of the object. The obstacle avoidance strategy is used to control the movement path generated by the mobile device.

[0052] According to some exemplary embodiments of the present disclosure, the obstacle avoidance strategy may be a specific strategy for the mobile device to avoid obstacles during movement. The movement path may be a path constructed by the mobile device during movement.

[0053] After identifying the types and locations of objects in the surrounding environment, the autonomous device can avoid obstacles based on the object type and generate a movement path. For example, a robot vacuum cleaner can identify objects to be cleaned and obstacles. During movement, if the robot vacuum recognizes an object to be cleaned (such as a stain), it will move to the object and begin cleaning. If it recognizes an obstacle (such as a scale), it will automatically avoid the obstacle and move around it.

[0054] The control method for a self-propelled device in this exemplary embodiment combines image information of the surrounding environment with laser point cloud information to determine object types, resulting in more accurate classification results and improved object recognition accuracy. Furthermore, the device's obstacle avoidance strategy is determined based on the object recognition results, making the subsequent generated device movement path more reasonable.

[0055] Next, the control method of the self-moving device in this exemplary embodiment will be further described.

[0056] In an exemplary embodiment of the present disclosure, for step S120, object recognition processing and result fusion processing are performed based on image information and laser point cloud information to obtain item type and position information of the item to be identified in the surrounding environment, including: performing object recognition processing based on image information and laser point cloud information to obtain an item type recognition result, the item type recognition result including an image recognition result and / or a point cloud recognition result; obtaining an image point cloud relationship between the image information and the laser point cloud information, the image point cloud relationship including a correspondence between pixel positions of the environment image and ranging points of the point cloud data; obtaining an item category point cloud and / or an item category image corresponding to the item type recognition result based on the image point cloud relationship; and performing result fusion processing on the item type recognition result with the corresponding item category point cloud and / or item category image to obtain item type and position information.

[0057] The object type recognition result may be the result obtained by identifying the specific type of the object to be identified in the surrounding environment. The image recognition result may be the type recognition result obtained by identifying the specific type of the object to be identified in the surrounding environment based on image information. The point cloud recognition result may be the item type recognition result obtained by identifying the specific type of the object to be identified in the surrounding environment based on laser point cloud information. The image-point cloud relationship may be the correspondence between the pixel positions of pixels in the environment image and the positions of ranging points in the three-dimensional point cloud data.

[0058] The environmental image may be an image corresponding to the surrounding environment. The pixel position may be the coordinates of each pixel in the environmental image. The point cloud data may be a dataset consisting of a three-dimensional point cloud constituting the surrounding environment. The ranging points may be all points contained in the point cloud data. The item category point cloud may be point cloud data corresponding to the image recognition result determined based on the image-point cloud relationship. The item category image may be composed of image pixels corresponding to the point cloud recognition result determined based on the image-point cloud relationship.

[0059] After obtaining the image information and laser point cloud information, object recognition processing can be performed based on at least one of the image information and the laser point cloud information to obtain an object type recognition result. The recognition result obtained from the object recognition processing based on the image information is used as the image recognition result; the recognition result obtained from the object recognition processing based on the laser point cloud information is used as the point cloud recognition result. The point cloud recognition result includes point cloud data corresponding to the object to be identified in the surrounding environment. The object recognition processing process is designed to identify objects in the surrounding environment, including balls of yarn, fans, electronic scales, and stains.

[0060] Self-driving devices can typically pre-calibrate RGB and ToF information, which in turn determines the correspondence between the two sensors, namely, the image point cloud relationship between the image information and the laser point cloud information. For example, the image information captured by the camera sensor is 10 cm in front of and 5 cm to the left of the self-driving device, while the laser point cloud information captured by the lidar device is 10 cm in front of and 5 cm to the right of the self-driving device. Therefore, the correspondence between the image information and the laser point cloud information is on the same horizontal plane, with a left-right position difference of 10 cm.

[0061] Based on the image point cloud relationship, the distance measurement point in the point cloud data corresponding to a certain pixel position in the image recognition result can be determined; and the image pixel corresponding to a certain distance measurement point in the point cloud recognition result can be determined. Therefore, when the item type recognition result is an image recognition result, the corresponding item category point cloud can be determined based on the image point cloud relationship; when the item type recognition result is a point cloud recognition result, the corresponding item category image can be determined based on the image point cloud relationship.

[0062] After obtaining the item category point cloud and / or item category image corresponding to the item type recognition result, the image recognition result and the corresponding item category point cloud are fused to determine the distance measurement points corresponding to each pixel in the image recognition result in the point cloud data, thereby identifying the item type in the surrounding environment and the location information of each item type in the surrounding environment. Alternatively, the point cloud recognition result and the corresponding item category image are fused to determine the image pixels corresponding to each distance measurement point in the point cloud recognition result, and then the item type and location information of the item to be identified are determined based on the fusion result. By performing type recognition processing on image information and laser point cloud information, not only can the image recognition result be obtained, but also the point cloud distance information can be obtained. The combination of the two can produce an item recognition result with higher accuracy.

[0063] In an exemplary embodiment of the present disclosure, object recognition processing is performed based on image information and laser point cloud information to obtain image recognition results and point cloud recognition results, including: performing object recognition processing based on image information to obtain an image recognition result of the object to be identified, the image recognition result including the object bounding box or segmented pixel information; and / or performing object recognition processing based on laser point cloud information to obtain a point cloud recognition result of the object to be identified, the point cloud recognition result including the object category point.

[0064] The object bounding box (Bounding Box), also known as a bounding volume or bounding region, is a rectangular box used to describe the location and range of an object in an image. Segmented pixel information can be obtained by classifying an image at the pixel level. Pixels belonging to the same category are grouped together. For example, pixels belonging to people are grouped together, pixels belonging to objects are grouped together, and pixels belonging to the background are grouped together. Item category points can be ranging points corresponding to items of the same category in the point cloud data.

[0065] Because image information includes the surrounding environment, the detection task of object recognition is performed on the environmental image. In this detection task, the model detects and locates the object by predicting the bounding box of the target object in the environmental image. The final output is the bounding box, which is defined by the coordinates of the upper left and lower right corners of the rectangular box. It is used to mark and locate the target object and facilitate its manipulation and analysis. In the image-based object detection task, the bounding box marked on the RGB image is used as the image recognition result.

[0066] In image semantic segmentation, the environment image is input into a fully convolutional neural network (FCN). In this network model, convolutional layers replace the fully connected layers in conventional convolutional neural networks (CNN). By fusing information at different scales, image segmentation maps of any size can be generated, thereby enabling pixel-level classification of the image. In image semantic segmentation, the segmented pixel information marked on the RGB image is used as the image recognition result.

[0067] Because the mobile device acquires laser point cloud information of its surroundings, it can also perform object recognition based on this information. Point cloud data corresponding to the surrounding environment is determined based on this information, and points within the point cloud data containing object recognition results are marked, including objects such as balls of yarn, fans, and electronic scales. These points are referred to as object classification points. Object recognition using image information and / or laser point cloud information can be used to subsequently determine the item type.

[0068] In an exemplary embodiment of the present disclosure, an item type recognition result is fused with a corresponding item category point cloud and / or item category image to obtain item type and location information, including: projecting the item bounding box or segmented pixel information onto the item category point cloud to obtain point cloud features of the item category point cloud; classifying, filtering, and identifying the image recognition result based on the point cloud features to obtain the item type, and determining the location information of the item type.

[0069] The item category point cloud can be three-dimensional point cloud data corresponding to different categories of items in the surrounding environment. Ranging points belonging to the same item are classified into the same item category point cloud. The point cloud features of the item category point cloud can be specific features contained in the point cloud data of the point cloud recognition results. For example, point cloud features can include spatial features and ranging features. Classification and filtering recognition can be a process that filters the item category classification results determined based on image information using point cloud features to eliminate inaccurate classification results.

[0070] Image recognition results can be expressed using object bounding boxes or segmented pixel information. Taking the object bounding box as an example, the point cloud data corresponding to the pixels contained in the TOF category object can be calculated based on the image point cloud relationship information. This is used as the item category point cloud to implement the subsequent image RGB information and ToF data alignment process. After the image RGB information and ToF data are aligned, the image recognition results are combined with the ToF depth information to form a sensor similar to a color depth map (RGBD), determining the type and location of objects in the surrounding environment for subsequent obstacle avoidance.

[0071] There are two main types of solutions that combine the two. One is to project the bounding box or pixel information in the image onto the object category point cloud to find the corresponding point cloud features. The other is to directly map the valid point cloud (point cloud data corresponding to the objects in the surrounding environment) to the image, and then invert the point cloud features based on the image box or segmentation results. After obtaining the point cloud features, the image recognition results are classified and filtered according to the point cloud features to obtain the object type and determine the location coordinates of each object type in the surrounding environment. By fusing the image recognition results with the point cloud recognition results and determining the object type, more accurate category information can be obtained.

[0072] In an exemplary embodiment of the present disclosure, image recognition results are classified, filtered and identified based on point cloud features to obtain the type of object, including: when the image recognition result is the first obstacle and the point cloud feature does not have height information, determining the object type as no obstacle; when the image recognition result is the first obstacle and the point cloud feature has height information, obtaining spatial attribute features of the point cloud data based on the point cloud feature; and determining the object type based on the spatial attribute features.

[0073] The height information may be related information of the height data used to characterize the point cloud data, and the spatial attribute feature may be a feature used to characterize the three-dimensional spatial attribute of the point cloud data.

[0074] After obtaining the image recognition results, further classification, filtering and identification are performed in combination with the point cloud features. When the image recognition result is the first obstacle, for example, the first obstacle is wire, for the wire in the environmental image, after finding the corresponding item category point cloud, more accurate distance information can be calculated based on the item category point cloud, thereby improving the obstacle avoidance effect. For example, when there is a ball of wire on the ground, the image result can be identified, and at the same time, a certain height can be measured on the item category point cloud, that is, the point cloud feature has height information. At this time, the spatial attribute features of the point cloud data will be obtained based on the point cloud features, and then further identification and determination of whether the item is wire will be made based on the spatial attribute features.

[0075] However, when there are patterns on the ground that resemble wire (such as floor patterns, carpet patterns, and floor mat patterns), the image may also identify this pattern as wire. However, the TOF point cloud does not contain clear height information for this pattern. In other words, the point cloud features do not have height information. In this case, the misidentified wire is filtered out and the object type is determined to be obstacle-free. At this time, the obstacle avoidance distance corresponding to the self-moving device is relatively close. Using point cloud features in the object type determination process can further filter out misidentified results in the image and improve the accuracy of the recognition results.

[0076] In an exemplary embodiment of the present disclosure, determining the type of an item based on a spatial attribute feature includes: when the spatial attribute feature is that the space is not closed, determining the type of the item as a first obstacle; when the spatial attribute feature is that the space is closed, determining the type of the item as a second obstacle.

[0077] Among them, spatial closure can be a feature that when light is irradiated on the object, the light cannot penetrate the entire surface of the object. Spatially non-closure can be a feature that when light is irradiated on the object, the light can penetrate part of the entire surface of the object and cannot penetrate part of the entire surface.

[0078] Refer to Figure 2, which shows a schematic diagram of identifying wire items according to an exemplary embodiment of the present disclosure. If the image recognition result is the first obstacle, and the point cloud feature has height information, after determining the spatial attribute characteristics of the three-dimensional point cloud data from the point cloud feature, the specific type of items in the surrounding environment is further determined in combination with the spatial attribute characteristics. If the spatial attribute characteristics of the point cloud data are spatially closed, the combination of the two can confirm that the wire 210 really exists. That is, when the self-mobile device really recognizes the wire, it will also adopt a corresponding obstacle avoidance strategy, such as the detour distance will be much farther than that of ordinary obstacles.

[0079] Refer to Figure 3, which shows a schematic diagram of identifying low obstacles according to an exemplary embodiment of the present disclosure. If the image recognition result is a second obstacle, such as a low obstacle, for other low obstacles, such as items such as scales, small toys, and power strips, there are sometimes misidentifications in the image. At this time, the TOF point cloud can also indirectly confirm whether the detection result is correct. For example, such small objects must have height information in TOF. When the point cloud feature has height information, the spatial attribute characteristics of the point cloud data are further judged to determine whether the space is closed, thereby shielding the erroneous recognition results on some planes. When the point cloud feature has height information and the spatial attribute feature is spatial closure, the item type is determined to be a low obstacle, such as a scale 310. Through the above-mentioned recognition method, more accurate recognition information can be obtained.

[0080] In an exemplary embodiment of the present disclosure, the image recognition results are classified, filtered and identified based on the point cloud features to obtain the object type, including: when the image recognition result is the third obstacle and the point cloud feature has height information, determining the object type as no obstacle; when the image recognition result is the third obstacle and the point cloud feature does not have height information, obtaining reflection intensity information based on the point cloud feature; and determining the object type based on the reflection intensity information.

[0081] The reflection intensity information may be related information used to characterize the reflection intensity of point cloud data.

[0082] Referring to Figure 4, a schematic diagram illustrating stain identification according to an exemplary embodiment of the present disclosure is shown. If the image recognition result indicates a third obstacle, such as a stain, the corresponding point cloud is found for the identified stain. Point cloud features are then acquired. If the point cloud features display significant height information, this indicates that the stain information found in the image is inaccurate. Stains are generally considered to have no height, and inaccurate identification results are filtered out. Additionally, the likelihood of stain 410 being detected can be determined based on the point cloud's intensity, reflectivity, and other information.

[0083] In TOF, there is an indicator that reflects the intensity of the reflection of the point cloud. For example, if the reflection of a metal object is relatively strong, the reflection intensity of the point cloud hitting the object will be higher. For black objects or objects made of light-absorbing materials (such as matte surfaces), the reflection is relatively weak, so the reflection intensity hitting the object is relatively weak. When the sweeping robot is cleaning, if a stain is detected on the image, and there is also an area on the point cloud with a reflection that is significantly different from other positions, for example, the reflection intensity on the floor material is average, but when a piece of soy sauce is sprinkled on it, the reflection in the soy sauce area will be much weaker than the surrounding area; or if a piece of juice is sprinkled on a material with weak reflection, such as a door sill, the reflection of the juice will be much stronger than the surrounding area, which further confirms that the stain detection is correct.

[0084] Conversely, if a stain is detected in the image but there is no significant change in the reflective information in the corresponding TOF area, it means that the stain may have been misidentified. The above classification and filtering recognition method will further filter out misidentified results in the image and obtain more accurate classification information.

[0085] In an exemplary embodiment of the present disclosure, step S120 involves fusing the item type recognition result with the corresponding item category point cloud and / or item category image to obtain item type and location information, including: aligning the point cloud recognition result with the item category image based on the image-point cloud relationship to obtain an initial fusion result, wherein the image pixels in the initial fusion result have ranging data; obtaining color features corresponding to the item category image; and filtering and identifying the initial fusion result based on the color features to obtain the item type and location information.

[0086] Position alignment can be the process of aligning the position coordinates of the point cloud recognition results with the pixels of the item classification image. The initial fusion result can be the item classification result obtained after the position alignment process. The color feature can be the feature corresponding to the RGB information contained in the item classification image corresponding to the point cloud recognition result. The filtering recognition process can be the process of filtering out misclassified results in the classification result.

[0087] This embodiment also provides another implementation scheme for fusing the two recognition results to determine the item type. Based on the image point cloud relationship, the point cloud recognition result and the item category image are aligned to obtain an initial fusion result. After the alignment operation, the point cloud contains RGB information, and image processing techniques can be used to filter out misidentification results. For example, the initial fusion result can be filtered using the color characteristics of the environment image to obtain the item type and location information.

[0088] For example, a metal box and a thick book may look similar based on the point cloud data, but the image shows the metal box to be brighter and more metallic, while the book is darker. This feature can be used to further verify the accuracy of the point cloud recognition result. For an object identified by point cloud, if its corresponding image lacks any possible color features, it can be used to filter out misidentified results. The color features here refer to the example of distinguishing between books and metal boxes based on their brightness. This type recognition scheme, based on point cloud data with color information, can further improve the accuracy of the recognition results.

[0089] In an exemplary embodiment of the present disclosure, based on the image-point cloud relationship, the point cloud recognition result and the item category image are positionally aligned to obtain an initial fusion result, including: based on the image-point cloud relationship, projecting the item category points in the point cloud recognition result to the item category image to obtain the initial fusion result; or based on the image-point cloud relationship, projecting the item category image to the point cloud data to obtain the initial fusion result.

[0090] During the position alignment process between the point cloud recognition results and the item category image, since the point cloud recognition results include point cloud data of the surrounding environment, and since RGB and ToF have been calibrated in advance, the approximate correspondence between the two sensors can be obtained, that is, the image point cloud relationship. Based on the image point cloud relationship, the point cloud recognition results and the item category image can be aligned, specifically including the following two categories:

[0091] One is to project the points with category information in the point cloud (i.e., item category points) onto the item category image to obtain the initial fusion result. The other is to align the item category image to the point cloud recognition result, even if the point cloud carries color information. The above processing process can be understood as first knowing the positional relationship between the point cloud and the image, that is, the installation position of the two, and then you can choose to project the point cloud result onto the item category image, that is, align the point cloud to the pixel coordinates so that each pixel has distance information. The item category image can also be projected onto the point cloud, that is, each point has a color, and one pixel may correspond to multiple point clouds. Through the above processing process, the image recognition result and the point cloud recognition result can be aligned in position, so that type recognition processing can be performed based on the aligned fusion result.

[0092] In an exemplary embodiment of the present disclosure, for step S120, object recognition processing and result fusion processing are performed based on image information and laser point cloud information to obtain the object type and location information of the object to be identified in the surrounding environment, including: performing object recognition processing based on image information and laser point cloud information to obtain image recognition results and point cloud recognition results; merging the image recognition results and point cloud recognition results to obtain the object type; obtaining the image-point cloud relationship, and determining the location information corresponding to the object type based on the image-point cloud relationship.

[0093] The result merging process may be a process of taking the union of the image recognition result and the point cloud recognition result.

[0094] In the above-mentioned result fusion scheme, object recognition can be performed using only image information or only laser point cloud information. In this embodiment, object recognition is performed based on both image information and laser point cloud information, respectively, to obtain corresponding image recognition results and point cloud recognition results. After obtaining both recognition results, data alignment is performed by projecting the image onto the point cloud, or vice versa, similar to the above process.

[0095] Because both image and point cloud recognition results contain category information, combining the two results, such as taking an intersection, can determine the object type. Once the object type is determined, the location coordinates of each object type within the surrounding environment are determined based on the image point cloud relationship. Since both contain category information, combining the two and taking their intersection can improve recognition accuracy.

[0096] In summary, the control method of the self-moving device disclosed herein obtains image information and laser point cloud information of the surrounding environment; performs object recognition processing and result fusion processing based on the image information and laser point cloud information to obtain the item type and location information of the item to be identified in the surrounding environment; and determines the obstacle avoidance strategy of the self-moving device based on the item type and location information, and the obstacle avoidance strategy is used to control the movement path generated by the self-moving device. On the one hand, combining the image information of the surrounding environment with the laser point cloud information to jointly determine the item type can obtain a more accurate classification result, thereby improving the accuracy of item recognition. On the other hand, determining the device obstacle avoidance strategy based on the item recognition result makes the subsequently generated device movement path more reasonable.

[0097] It should be noted that although the steps of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all steps must be performed to achieve the desired results. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0098] In addition, in this exemplary embodiment, a control device for a self-moving device is also provided. Referring to FIG. 5 , the control device 500 for a self-moving device may include: an environment information acquisition module 510 , an item type determination module 520 , and a device control module 530 .

[0099] Specifically, the environmental information acquisition module 510 is used to obtain image information and laser point cloud information of the surrounding environment; the object type determination module 520 is used to perform object recognition processing and result fusion processing based on the image information and laser point cloud information to obtain the object type and location information of the object to be identified in the surrounding environment; the device control module 530 is used to determine the obstacle avoidance strategy of the mobile device based on the object type and location information, and the obstacle avoidance strategy is used to control the movement path generated by the mobile device.

[0100] In an exemplary embodiment of the present disclosure, the item type determination module 520 includes a first type determination unit, which is used to: perform item recognition processing based on image information and laser point cloud information to obtain an item type recognition result, where the item type recognition result includes an image recognition result and / or a point cloud recognition result; obtain an image point cloud relationship between the image information and the laser point cloud information, where the image point cloud relationship includes a correspondence between pixel positions of the environment image and ranging points of the point cloud data; obtain an item category point cloud and / or an item category image corresponding to the item type recognition result based on the image point cloud relationship; and perform result fusion processing on the item type recognition result and the corresponding item category point cloud and / or item category image to obtain item type and position information.

[0101] In an exemplary embodiment of the present disclosure, the first type determination unit includes a type recognition subunit, which is used to: perform object recognition processing based on image information to obtain an image recognition result of the object to be identified, and the image recognition result includes the object bounding box or segmented pixel information; and / or perform object recognition processing based on laser point cloud information to obtain a point cloud recognition result of the object to be identified, and the point cloud recognition result includes an object category point.

[0102] In an exemplary embodiment of the present disclosure, the image recognition result includes an object bounding box or segmented pixel information, and the first type determination unit includes a first type determination subunit, which is used to: project the object bounding box or segmented pixel information onto an object category point cloud to obtain point cloud features of the object category point cloud; classify, filter and identify the image recognition result according to the point cloud features to obtain the object type, and determine the location information of the object type.

[0103] In an exemplary embodiment of the present disclosure, the first type determination subunit includes a first filtering subunit, which is used to: when the image recognition result is the first obstacle and the point cloud feature does not have height information, determine the object type as no obstacle; when the image recognition result is the first obstacle and the point cloud feature has height information, obtain the spatial attribute characteristics of the point cloud data based on the point cloud feature; and determine the object type according to the spatial attribute characteristics.

[0104] In an exemplary embodiment of the present disclosure, the first type determination subunit includes a second filtering subunit, which is used to: when the spatial attribute feature is that the space is not closed, determine the item type as a first obstacle; when the spatial attribute feature is that the space is closed, determine the item type as a second obstacle.

[0105] In an exemplary embodiment of the present disclosure, the first type determination subunit includes a third filtering subunit, which is used to: when the image recognition result is the third obstacle and the point cloud feature has height information, determine the object type as no obstacle; when the image recognition result is the third obstacle and the point cloud feature does not have height information, obtain reflection intensity information based on the point cloud feature; and determine the object type according to the reflection intensity information.

[0106] In an exemplary embodiment of the present disclosure, the item type determination module 520 includes a second type determination unit, which is used to: based on the image point cloud relationship, align the point cloud recognition result with the item category image to obtain an initial fusion result, in which the image pixels in the initial fusion result have ranging data; obtain color features corresponding to the item category image; and filter and identify the initial fusion result based on the color features to obtain item type and location information.

[0107] In an exemplary embodiment of the present disclosure, the second type determination unit includes a fusion processing subunit, which is used to: project the item category points in the point cloud recognition result to the item category image based on the image-point cloud relationship to obtain an initial fusion result; or project the item category image to the point cloud recognition result based on the image-point cloud relationship to obtain an initial fusion result.

[0108] In an exemplary embodiment of the present disclosure, the item type determination module 520 includes a third type determination unit, which is used to: perform item recognition processing based on image information and laser point cloud information to obtain image recognition results and point cloud recognition results; merge the image recognition results and point cloud recognition results to obtain the item type; obtain the image point cloud relationship, and determine the location information corresponding to the item type based on the image point cloud relationship.

[0109] The specific details of the virtual modules of the control devices of the above-mentioned respective mobile devices have been described in detail in the control methods of the corresponding mobile devices, and thus will not be repeated here.

[0110] The technical solution provided by the present disclosure may have the following beneficial effects:

[0111] The control method of the self-moving device in the exemplary embodiment of the present disclosure, on the one hand, combines the image information of the surrounding environment with the laser point cloud information to jointly determine the type of object, which can obtain more accurate classification results and improve the accuracy of object identification. On the other hand, the device obstacle avoidance strategy is determined based on the object identification results, so that the subsequently generated device movement path is more reasonable. It should be noted that although several modules or units of the control device of the self-moving device are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided to be concretized by multiple modules or units.

[0112] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0113] Those skilled in the art will appreciate that various aspects of the present invention may be implemented as systems, methods, or program products. Accordingly, various aspects of the present invention may be implemented as a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or a combination of hardware and software embodiments, which may be collectively referred to herein as "circuits," "modules," or "systems."

[0114] The electronic device 600 according to this embodiment of the present disclosure is described below with reference to Figure 6. The electronic device 600 shown in Figure 6 is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0115] As shown in FIG6 , electronic device 600 is implemented as a general-purpose computing device. Components of electronic device 600 may include, but are not limited to, the aforementioned at least one processing unit 610, the aforementioned at least one storage unit 620, a bus 630 connecting various system components (including storage unit 620 and processing unit 610), and a display unit 640.

[0116] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0117] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 621 and / or a cache memory unit 622 , and may further include a read-only memory unit (ROM) 623 .

[0118] The storage unit 620 may include a program / utility 624 having a set (at least one) of program modules 625, such program modules 625 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0119] Bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0120] The electronic device 600 can also communicate with one or more external devices 670 (e.g., a keyboard, a pointing device, a Bluetooth device, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., a router, a modem, etc.). Such communication can occur via an input / output (I / O) interface 650. Furthermore, the electronic device 600 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 660. As shown, the network adapter 660 communicates with other modules of the electronic device 600 via a bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 600, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0121] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0122] In exemplary embodiments of the present disclosure, a computer-readable storage medium is also provided, storing a program product capable of implementing the aforementioned methods of this specification. In some possible embodiments, various aspects of the present invention may also be implemented in the form of a program product comprising program code. When the program product is executed on a terminal device, the program code is configured to cause the terminal device to perform the steps according to various exemplary embodiments of the present invention described in the "Exemplary Methods" section of this specification.

[0123] Referring to FIG7 , a program product 700 for implementing the above method according to an embodiment of the present invention is described. The program product 700 may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0124] The program product may be implemented in any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0125] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0126] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0127] The program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0128] Furthermore, the above-described figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention and are not intended to be limiting. It is readily understood that the processes illustrated in the above-described figures do not indicate or limit the temporal order of these processes. Furthermore, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0129] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow from the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the claims.

[0130] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for controlling a self-propelled device, comprising: Obtain image information and laser point cloud information of the surrounding environment; Performing object recognition processing and result fusion processing based on the image information and the laser point cloud information to obtain object type and location information of the object to be recognized in the surrounding environment; An obstacle avoidance strategy for the self-moving device is determined according to the item type and the location information, and the obstacle avoidance strategy is used to control and generate a moving path for the self-moving device.

2. The method according to claim 1, wherein The object recognition processing and result fusion processing based on the image information and the laser point cloud information to obtain the object type and location information of the object to be recognized in the surrounding environment includes: Performing object recognition processing based on the image information and / or the laser point cloud information to obtain an object type recognition result, wherein the object type recognition result includes an image recognition result and / or a point cloud recognition result; Acquire an image point cloud relationship between the image information and the laser point cloud information, wherein the image point cloud relationship includes a correspondence between pixel positions of the environment image and ranging points of the point cloud data; Based on the image point cloud relationship, obtaining an item category point cloud and / or an item category image corresponding to the item type recognition result; The item type recognition result is subjected to the result fusion processing with the corresponding item category point cloud and / or the item category image to obtain the item type and the location information.

3. The method according to claim 2, wherein: The performing of object recognition processing based on the image information and / or the laser point cloud information to obtain an object type recognition result includes: Performing object recognition processing based on the image information to obtain an image recognition result of the object to be recognized, the image recognition result including object boundary box or segmented pixel information; and / or Object recognition processing is performed based on the laser point cloud information to obtain a point cloud recognition result of the object to be recognized, wherein the point cloud recognition result includes an object category point.

4. The method according to claim 2, wherein: The image recognition result includes an item bounding box or segmented pixel information, and fusing the item type recognition result with the corresponding item category point cloud and / or the item category image to obtain the item type and location information includes: Projecting the object bounding box or the segmented pixel information onto the object category point cloud to obtain point cloud features of the object category point cloud; The image recognition result is classified, filtered and identified according to the point cloud features to obtain the item type and determine the location information of the item type.

5. The method according to claim 4, wherein The classifying, filtering and identifying the image recognition result according to the point cloud features to obtain the item type includes: When the image recognition result is a first obstacle and the point cloud feature does not have height information, the object type is determined to be no obstacle; When the image recognition result is a first obstacle and the point cloud feature has height information, obtaining a spatial attribute feature of the point cloud data based on the point cloud feature; The item type is determined according to the spatial attribute characteristics.

6. The method according to claim 5, wherein: The determining the item type according to the spatial attribute feature includes: When the spatial attribute characteristic is that the space is not closed, determining the object type as the first obstacle; When the spatial attribute feature is closed space, the object type is determined to be a second obstacle.

7. The method according to claim 4, wherein: The classifying, filtering and identifying the image recognition result according to the point cloud features to obtain the item type includes: When the image recognition result is a third obstacle and the point cloud feature has height information, the object type is determined to be no obstacle; When the image recognition result is a third obstacle and the point cloud feature does not have height information, obtaining reflection intensity information based on the point cloud feature; The type of the object is determined according to the reflection intensity information.

8. The method according to claim 2, wherein: The fusing the item type recognition result with the corresponding item category point cloud and / or the item category image to obtain the item type and location information includes: Based on the image point cloud relationship, position alignment processing is performed on the point cloud recognition result and the item category image to obtain an initial fusion result, where the image pixels in the initial fusion result have ranging data; Obtaining color features corresponding to the item category image; The initial fusion result is filtered and identified based on the color feature to obtain the item type and the location information.

9. The method according to claim 8, wherein The step of aligning the point cloud recognition result with the item category image based on the image point cloud relationship to obtain an initial fusion result includes: Based on the image point cloud relationship, projecting the item category points in the point cloud recognition result to the item category image to obtain the initial fusion result; or Based on the image-point cloud relationship, the item category image is projected onto the point cloud recognition result to obtain the initial fusion result.

10. The method according to claim 1, wherein The object recognition processing and result fusion processing based on the image information and the laser point cloud information to obtain the object type and location information of the object to be recognized in the surrounding environment includes: Performing object recognition processing based on the image information and the laser point cloud information to obtain an image recognition result and a point cloud recognition result; Merging the image recognition result and the point cloud recognition result to obtain the object type; Obtain an image point cloud relationship, and determine the location information corresponding to the item type based on the image point cloud relationship.

11. A control device for a mobile device, comprising: Environmental information acquisition module, used to obtain image information and laser point cloud information of the surrounding environment; An object type determination module is configured to perform object identification processing and result fusion processing based on the image information and the laser point cloud information to obtain object type and location information of the object to be identified in the surrounding environment; The device control module is used to determine an obstacle avoidance strategy for the self-moving device according to the item type and the location information, wherein the obstacle avoidance strategy is used to control the generation of a moving path for the self-moving device.

12. The device according to claim 11, wherein The item type determination module includes a first type determination unit, which is configured to: Performing object recognition processing based on the image information and / or the laser point cloud information to obtain an object type recognition result, wherein the object type recognition result includes an image recognition result and / or a point cloud recognition result; Acquire an image point cloud relationship between the image information and the laser point cloud information, wherein the image point cloud relationship includes a correspondence between pixel positions of the environment image and ranging points of the point cloud data; Based on the image point cloud relationship, obtaining an item category point cloud and / or an item category image corresponding to the item type recognition result; The item type recognition result is subjected to the result fusion processing with the corresponding item category point cloud and / or the item category image to obtain the item type and the location information.

13. The device according to claim 12, wherein The first type determination unit includes a type identification subunit, and the type identification subunit is configured to: Performing object recognition processing based on the image information to obtain an image recognition result of the object to be recognized, the image recognition result including an object boundary box or segmented pixel information; and / or Object recognition processing is performed based on the laser point cloud information to obtain a point cloud recognition result of the object to be recognized, wherein the point cloud recognition result includes an object category point.

14. The device according to claim 12, wherein The image recognition result includes an object bounding box or segmented pixel information, and the first type determination unit includes a first type determination subunit, which is configured to: Projecting the object bounding box or the segmented pixel information onto the object category point cloud to obtain point cloud features of the object category point cloud; The image recognition result is classified, filtered and identified according to the point cloud features to obtain the item type and determine the location information of the item type.

15. The device according to claim 14, wherein The first type determination subunit includes a first filtering subunit, and the first filtering subunit is configured to: When the image recognition result is a first obstacle and the point cloud feature does not have height information, the object type is determined to be no obstacle; When the image recognition result is a first obstacle and the point cloud feature has height information, obtaining a spatial attribute feature of the point cloud data based on the point cloud feature; The item type is determined according to the spatial attribute characteristics.

16. The device according to claim 15, wherein The first type determination subunit includes a second filtering subunit, and the second filtering subunit is configured to: When the spatial attribute characteristic is that the space is not closed, determining the object type as the first obstacle; When the spatial attribute feature is closed space, the object type is determined to be a second obstacle.

17. The device according to claim 14, wherein The first type determination subunit includes a third filtering subunit, and the third filtering subunit is configured to: When the image recognition result is a third obstacle and the point cloud feature has height information, the object type is determined to be no obstacle; When the image recognition result is a third obstacle and the point cloud feature does not have height information, obtaining reflection intensity information based on the point cloud feature; The type of the object is determined according to the reflection intensity information.

18. The device according to claim 12, wherein The item type determination module includes a second type determination unit, wherein the second type determination unit is configured to: Based on the image point cloud relationship, position alignment processing is performed on the point cloud recognition result and the item category image to obtain an initial fusion result, where the image pixels in the initial fusion result have ranging data; Obtaining color features corresponding to the item category image; The initial fusion result is filtered and identified based on the color feature to obtain the item type and the location information.

19. The device according to claim 18, wherein The second type determination unit includes a fusion processing subunit, and the fusion processing subunit is configured to: Based on the image point cloud relationship, projecting the item category points in the point cloud recognition result to the item category image to obtain the initial fusion result; or Based on the image-point cloud relationship, the item category image is projected onto the point cloud recognition result to obtain the initial fusion result.

20. The device according to claim 11, wherein The item type determination module includes a third type determination unit, and the third type determination unit is configured to: Performing object recognition processing based on the image information and the laser point cloud information to obtain an image recognition result and a point cloud recognition result; Merging the image recognition result and the point cloud recognition result to obtain the object type; Obtain an image point cloud relationship, and determine the location information corresponding to the item type based on the image point cloud relationship.

21. An electronic device comprising: processor; as well as A memory having computer-readable instructions stored thereon, wherein the computer-readable instructions, when executed by the processor, implement the control method of the self-moving device according to any one of claims 1 to 10.

22. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the control method of the self-mobile device according to any one of claims 1 to 10 is implemented.

23. A computer program product comprising a computer program, wherein When the computer program is executed by a processor, the control method of the self-moving device according to any one of claims 1 to 10 is implemented.

Images