Self-propelled device movement control method, device and self-propelled device

Abstract

The present application provides a method, device, electronic device, computer storage medium and self-propelled device for controlling the movement of a self-propelled device. In this method, a regional image of the area to be analyzed and a depth image of the area to be analyzed are obtained, and based on the regional image, it is detected whether the area contains an object of interest. If so, based on the object of interest, the regional image and the depth image of the area to be analyzed, an image of the object of interest and a depth image of the object of interest are obtained; then, it is determined whether the object of interest is a first object of interest; if so, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined. Based on this method, it is possible to accurately determine whether the object of interest is the first object of interest, so that objects of similar types among the objects of interest can be accurately distinguished. When it is determined that the object of interest is the first object of interest, a mutual avoidance strategy between the first object of interest and the self-propelled device is determined, thereby ensuring the safety of the first object of interest while avoiding the self-propelled device from taking a detour.

Description

Technical Field

[0001] The present application relates to the field of cleaning equipment, and specifically to a method and device for controlling the movement of a self-propelled device, an electronic device, a computer storage medium, and a self-propelled device. Background Art

[0002] With the continuous improvement of the quality of life, cleaning robots are becoming more and more popular. At the same time, the application scenarios of cleaning robots are also increasing. For example, cleaning robots can be used in ordinary households, offices, shopping malls or supermarkets to clean or sweep the environment.

[0003] Regardless of the scenario in which a cleaning robot is used, it must identify obstacles and people in the cleaning environment during operation. Some obstacles, such as wires, socks, or small toys, require the robot to navigate around them to avoid being sucked into the robot and damaging it. Other obstacles, such as walls, table and chair legs, or sofa legs, require the robot to gently touch them to clean the immediate surroundings or the ground. Of course, for people, the robot generally needs to navigate around them to avoid knocking them over. The obstacles listed above are easy for a cleaning robot to distinguish. Once the robot has identified these obstacles, it can avoid them using appropriate avoidance strategies, such as bypassing, gently touching, or climbing over them. However, when obstacles in the cleaning scene closely resemble a human figure, the robot may have difficulty distinguishing between people and obstacles, which can lead to incorrect avoidance strategies. Therefore, how to enable cleaning robots to accurately identify object types and avoid obstacles in such scenarios has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] This application provides a method for controlling the movement of a self-propelled device to solve the technical problem of how to enable a cleaning robot to accurately identify object types and thus accurately avoid obstacles. This application also provides a device, electronic equipment, and computer storage medium corresponding to the method. Furthermore, this application provides a self-propelled device.

[0005] The present application provides a method for controlling movement of a self-propelled device, comprising:

[0006] Acquire a regional image and a depth image of the area to be analyzed;

[0007] Performing object detection on the regional image to determine whether the detected objects include the object of interest;

[0008] If the detected objects include an object of interest, obtaining an image of the object of interest and a depth image of the object of interest based on the object of interest, the area image, and the depth image of the area to be analyzed;

[0009] determining, based on the object of interest image and the depth image of the object of interest, whether the object of interest is a first object of interest;

[0010] If the object of interest is a first object of interest, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined, where the avoidance strategy is used to control the self-propelled device to move.

[0011] Optionally, determining a mutual avoidance strategy between the self-propelled device and the first object of interest includes:

[0012] Controlling the self-propelled device to give a voice reminder to avoid the first object of interest;

[0013] If the first object of interest is not avoided within a preset time, the travel path of the self-propelled device is replanned to avoid the first object of interest.

[0014] Optionally, the method further includes: if the object of interest is a second object of interest, determining a movement control strategy for the self-propelled device to avoid the second object of interest.

[0015] Optionally, determining a mutual avoidance strategy between the self-propelled device and the first object of interest includes:

[0016] It is determined whether the first object of interest is on a path of the self-propelled device. If so, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined.

[0017] Optionally, determining a mutual avoidance strategy between the self-propelled device and the first object of interest includes:

[0018] It is determined whether the distance between the first object of interest and the self-propelled device is less than a preset first distance threshold. If so, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined.

[0019] Optionally, performing object detection on the regional image to determine whether the detected objects include the object of interest includes:

[0020] Using the region image as input data of an image object detection model to obtain an object detection result of the region image, wherein the image object detection model is a model for obtaining an object detection result of an image based on an image;

[0021] Based on the object detection result of the area image, it is determined whether the detected objects include the object of interest.

[0022] Optionally, obtaining the object of interest image and the depth image of the object of interest based on the object of interest, the region image, and the depth image of the area to be analyzed includes:

[0023] According to the object of interest, the region image is cropped to obtain an image of the object of interest;

[0024] According to the object of interest, the depth image of the object of interest is filtered in the depth image of the area to be analyzed.

[0025] Optionally, the depth image of the area to be analyzed is acquired by using a depth sensor installed on the self-propelled device.

[0026] Optionally, the determining whether the object of interest is the first object of interest based on the object of interest image and the depth image of the object of interest includes:

[0027] Using the image of the object of interest and the depth image of the object of interest as input data of a target convolutional neural network model to obtain attribute feature information of the object of interest, wherein the target convolutional neural network model is a model for obtaining object attribute feature information of an image based on an image;

[0028] According to the attribute characteristic information of the object of interest, it is determined whether the object of interest is the first object of interest.

[0029] Optionally, determining a movement control strategy for the self-propelled device to avoid the second object of interest includes:

[0030] The travel path of the self-propelled device is replanned to determine a replanned path; the replanned path is used to control the self-propelled device to move.

[0031] Optionally, the avoidance strategy includes at least one of the following:

[0032] Prompt the first concerned object to avoid the target through voice prompting;

[0033] Controlling the self-propelled device to slow down or stop moving;

[0034] Changing the travel path of the self-propelled device.

[0035] The present application provides a self-propelled device movement control device, comprising:

[0036] An original image acquisition unit, configured to acquire a regional image and a depth image of the region to be analyzed;

[0037] a first judging unit, configured to perform object detection on the region image and judge whether the detected objects include an object of interest;

[0038] an image of interest obtaining unit, configured to obtain an image of the object of interest and a depth image of the object of interest based on the object of interest, the region image, and the depth image of the region to be analyzed, if the detected objects include the object of interest;

[0039] a second determining unit, configured to determine whether the object of interest is a first object of interest based on the object of interest image and the depth image of the object of interest;

[0040] The mutual avoidance strategy determining unit is configured to determine a mutual avoidance strategy between the self-propelled device and the first object of interest if the object of interest is the first object of interest, wherein the avoidance strategy is used to control the self-propelled device to move.

[0041] The present application provides an electronic device, including:

[0042] processor;

[0043] The memory is used to store a computer program, which is executed by the processor to implement the movement control method of the self-propelled device.

[0044] The present application provides a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed by a processor to execute a method for controlling movement of a self-propelled device.

[0045] The present application provides a self-propelled device, comprising: an image acquisition device, a depth sensor, and a processor;

[0046] The image acquisition device is used to acquire a regional image of the area to be analyzed;

[0047] The depth sensor is used to obtain a depth image of the area to be analyzed;

[0048] The processor is used to receive the regional image of the area to be analyzed transmitted by the image acquisition device and the depth image of the area to be analyzed transmitted by the depth sensor, and perform object detection on the regional image to determine whether the detected objects include the object of interest; if the detected objects include the object of interest, obtain the object of interest image and the depth image of the object of interest based on the object of interest, the regional image and the depth image of the area to be analyzed; based on the object of interest image and the depth image of the object of interest, determine whether the object of interest is the first object of interest; if the object of interest is the first object of interest, determine a mutual avoidance strategy between the self-propelled device and the first object of interest, and the avoidance strategy is used to control the movement of the self-propelled device.

[0049] Compared with the prior art, the embodiments of the present application have the following advantages:

[0050] The present application provides a method for controlling the movement of a self-propelled device. In this method, a regional image and a depth image of a region to be analyzed are first obtained. Based on the regional image of the region to be analyzed, it is detected whether the region to be analyzed contains an object of interest. If the object of interest is contained, an image of the object of interest and a depth image of the object of interest are obtained based on the object of interest, the regional image, and the depth image of the region to be analyzed. Based on the image of the object of interest and the depth image of the object of interest, it is determined whether the object of interest is a first object of interest. If the object of interest is a first object of interest, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined, and the avoidance strategy is used to control the movement of the self-propelled device. In fact, after the method detects that the region to be analyzed contains an object of interest, based on the regional image, the image of the object of interest and the depth image of the object of interest are obtained based on the object of interest, the regional image, and the depth image of the region to be analyzed. Based on the depth image of the object of interest, it is possible to accurately determine whether the object of interest is the first object of interest. In this way, objects of similar types among the objects of interest can be accurately distinguished. When the object of interest is determined to be the first object of interest, a mutual avoidance strategy between the first object of interest and the self-propelled device is determined. Thus, while avoiding the self-propelled device from wasting energy by taking a detour, it is ensured that the self-propelled device and the first object of interest do not collide with each other, thereby ensuring the safety of the first object of interest.

[0051] In a further technical solution, since the first object of interest and the self-propelled device can avoid each other, when the first object of interest avoids the self-propelled device, the self-propelled device can avoid taking a detour; when the first object of interest does not avoid the self-propelled device, the self-propelled device chooses to avoid the first object of interest, so that the self-propelled device will not collide with the first object of interest, thereby ensuring the safety of the first object of interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0053] Figure 1 Flowchart of the self-propelled device movement control method provided in the first embodiment of the present application;

[0054] Figure 2 This is a schematic diagram of detecting whether an object of interest is contained in a region to be analyzed based on a regional image in the first embodiment of the present application;

[0055] Figure 3 This is a schematic diagram of determining whether an object of interest is a first object of interest based on an image of the object of interest and a depth image of the object of interest in the first embodiment of the present application;

[0056] Figure 4 A schematic diagram of a self-propelled device movement control device provided in a second embodiment of the present application;

[0057] Figure 5 This is a schematic diagram of an electronic device provided in the third embodiment of the present application. DETAILED DESCRIPTION

[0058] The following description sets forth many specific details to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of the present application. Therefore, the present application is not limited to the specific implementations disclosed below.

[0059] The present application provides a method, device, electronic device, computer storage medium and self-propelled device for controlling the movement of a self-propelled device. The following uses specific embodiments to describe the method, device, electronic device, computer storage medium and self-propelled device for controlling the movement of a self-propelled device of the present application. It should be noted that the embodiments described below are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0060] First embodiment

[0061] The first embodiment of the present application provides a method for controlling the movement of a self-propelled device. Figure 1 Please refer to Figure 1 , which is a flow chart of the self-propelled device movement control method provided in the first embodiment of the present application.

[0062] The method for controlling movement of a self-propelled device according to an embodiment of the present application includes the following steps.

[0063] Step S101: Acquire a regional image and a depth image of the area to be analyzed.

[0064] In this embodiment, the method for controlling the movement of a self-propelled device is applied to enable the self-propelled device to overcome various obstacles during its movement or motion. The self-propelled device may be a self-propelled robot in the field of cleaning equipment. This embodiment uses a cleaning robot as an example. It is understood that the self-propelled device may also be an autonomously movable device used in other fields, all of which fall within the scope of protection of this application.

[0065] In this embodiment, an image acquisition device and a depth sensor are mounted on the self-propelled device. The image acquisition device can acquire a regional image of the area to be analyzed in real time, and the depth sensor can acquire a depth image of the area to be analyzed in real time. By way of example, the regional image of the area to be analyzed can be an RGB image, and the depth sensor can include an RGBD camera. In practice, both the regional image and the depth image of the area to be analyzed can be acquired using an RGBD camera, meaning that the RGBD camera serves as both an image acquisition device and a depth sensor.

[0066] The area to be analyzed can be the area surrounding the autonomous vehicle's operating or traveling environment. For example, it can be a circle centered on the autonomous vehicle and with a predetermined radius. The purpose of acquiring both a regional image and a depth image of the area to be analyzed is to identify different types of objects within the area based on these images, thereby enabling obstacle avoidance during the autonomous vehicle's travel and ensuring the safety of the autonomous vehicle, other objects, and pedestrians.

[0067] Generally speaking, the regional image of the area to be analyzed obtained in this step can be a two-dimensional image, for example, an RGB image. Specifically, when the image acquisition device installed on the self-propelled device is a camera for capturing two-dimensional images, the two-dimensional image can be acquired using the camera.

[0068] The regional image can be used to preliminarily identify various objects in the area to be analyzed. In order to further accurately identify more similar objects or pedestrians in the area to be analyzed, the depth information of more similar objects or pedestrians is obtained through the depth image of the area to be analyzed to distinguish more similar objects or pedestrians.

[0069] Specifically, in order to obtain a depth image of the area to be analyzed, it can be achieved through a depth sensor installed on the self-propelled device. In this embodiment, the depth sensor is a depth camera installed on the self-propelled device. The depth camera is used to take real-time photos of possible objects in the area to be analyzed in the working environment or walking environment of the self-propelled device, thereby obtaining a depth image of the objects appearing in the area to be analyzed, and then obtaining the depth information of the objects.

[0070] More specifically, a depth camera can be installed in front of the cleaning robot. The depth camera of this embodiment can be a TOF, structured light, RGBD or other type of depth camera. In other words, the depth image captured by the depth camera only needs to be able to obtain the depth information of objects around the cleaning robot's working environment relative to the cleaning robot. For example, it can be the distance information between the object and the cleaning robot, or the specific three-dimensional spatial information of the object. That is, the distance between the cleaning robot and the obstacle can be obtained through the depth image, and the three-dimensional spatial structure information of the obstacle (such as height, width, length or thickness, etc.) can be obtained.

[0071] As an example, in this embodiment, an RGBD image is mainly used as a depth image for description.

[0072] Step S102: performing object detection on the region image to determine whether the detected objects include the object of interest.

[0073] In reality, a cleaning robot's working environment may present a variety of obstacles. For example, in a shopping mall, there may be many pedestrians and objects in the robot's working environment. In this embodiment, the focus is on distinguishing pedestrians or pedestrian-like objects, and then determining the obstacle avoidance strategy for the autonomous vehicle based on the recognition results. Whether pedestrians or other objects, they can be considered as research objects in the area to be analyzed in the area image.

[0074] In this embodiment, pedestrians and objects similar to pedestrians in the area to be analyzed are taken as objects of interest. Objects similar to pedestrians include human-shaped standees or large human-shaped dolls. Of course, it is understandable that multiple objects with similar appearances can also be taken as objects of interest to accurately identify these objects with similar appearances.

[0075] In this embodiment, object detection is performed on a regional image to determine whether the detected objects include the object of interest. The following method can be used: first, the regional image is used as input data of an image object detection model to obtain the object detection result of the regional image. The image object detection model is a model for obtaining the object detection result of an image based on an image; then, based on the object detection result of the regional image, it is determined whether the detected objects include the object of interest.

[0076] Specifically, the above-mentioned image object detection model is a trained deep learning network model that can identify each object in an image based on the image, such as a YOLO (you only look once) network. By inputting the image into the image object detection model, all objects in the image can be obtained.

[0077] As a way to obtain the above-mentioned image object detection model, the initial deep learning network model can be trained. Specifically, the process of training the initial deep learning network model is as follows.

[0078] First, a first image sample is obtained for training an initial deep learning network model. Then, the initial deep learning network model is trained using the first image sample until the trained deep learning network model can accurately identify objects in the image.

[0079] Specifically, the above-mentioned first image samples can be images in a shopping mall scene, and the objects in these first image samples are known. When the initial deep learning network model is trained, the first image samples are used as input data of the initial deep learning network model, and the object information corresponding to the first image samples is used as the output result to adjust the parameters in the initial deep learning network model. By using a large number of first image samples to train it and continuously adjusting the parameters in the initial deep learning network model, an image object detection model is finally obtained.

[0080] After obtaining the above-mentioned image object detection model, the image object detection model can be used to identify objects in the area image, and then obtain the object detection result of the area image. For example, when the area image taken by the cleaning robot contains a human-shaped standee of person A, pedestrian B and a box, the object detection result of the area image must also include the human-shaped standee of person A, pedestrian B and a box.

[0081] Then, based on the object detection results of the regional image, it is determined whether the detected objects include the object of interest. For example, if an object with a humanoid structure is considered the object of interest, then since the detected objects include the humanoid standee containing person A and pedestrian B, both of which have humanoid structures, it is determined that the detected objects include the object of interest. In other words, the humanoid standee containing person A and pedestrian B can be considered the objects of interest.

[0082] In this embodiment, since the human-shaped standee of person A and pedestrian B both contain human-shaped structures, it is impossible to accurately identify whether these objects are real people or "fake" people in the object detection results, and further distinction is required.

[0083] Since this embodiment is primarily used to identify objects with similar appearance and structure, in order to reduce the subsequent recognition workload and thus improve recognition efficiency, objects with similar appearance can be designated as focus objects, and other objects other than the focus objects will not be recognized in the subsequent refined recognition process. In a shopping mall scenario, objects with human-like structures can be directly designated as focus objects.

[0084] For the process of object detection on a region image and determining whether the detected object contains the object of interest, please refer to Figure 2 , which is a schematic diagram of detecting whether the area to be analyzed contains an object of interest based on a regional image in the first embodiment of the present application.

[0085] For example, two region images, Image 1 (the image of pedestrian B) and Image 2 (the image of a humanoid standee of person A), are fed into the YOLO network for humanoid structure detection. When both Image 1 and Image 2 are detected to contain humanoid structures, the humanoid portion (i.e., the object of interest) is identified and marked. The reason for identifying and marking the humanoid portion is to facilitate the subsequent acquisition of the object of interest image based on the marked object of interest.

[0086] Step S103: If the detected objects include an object of interest, an image of the object of interest and a depth image of the object of interest are obtained based on the object of interest, the region image and the depth image of the region to be analyzed.

[0087] After detecting the object of interest based on the region image, an image of the object of interest and a depth image of the object of interest are obtained based on the object of interest, the region image, and the depth image of the region to be analyzed. The purpose of obtaining the image of the object of interest and the depth image of the object of interest is to subsequently accurately identify different objects within the object of interest, for example, to distinguish between real people and "fake" people. It should be noted that in this embodiment, the image of the object of interest and the depth image of the object of interest have a corresponding relationship. That is, for each pixel in the image of the object of interest, its depth data can be found in the depth image of the object of interest. Of course, the region image of the region to be analyzed and the depth image of the region to be analyzed also have this corresponding relationship.

[0088] Specifically, obtaining the image of the object of interest and the depth image of the area to be analyzed based on the object of interest, the area image and the depth image of the area to be analyzed may mean: first, cropping the area image according to the object of interest to obtain the image of the object of interest; and at the same time, filtering the depth image of the object of interest in the depth image of the area to be analyzed according to the object of interest.

[0089] For the image of the object of interest obtained after cropping image 1 and image 2, please see Figure 3 , which is a schematic diagram of determining whether the object of interest is the first object of interest based on the image of the object of interest and the depth image of the object of interest in the first embodiment of the present application, and Figure 3 A depth image of the object of interest is also included.

[0090] At the same time, based on the depth images of the object of interest and the area to be analyzed, the depth image of the object of interest is filtered from the depth image of the area to be analyzed. After determining the object of interest, the depth image of the object of interest can be filtered from the depth image of the area to be analyzed. For example, the depth image of the area to be analyzed can be filtered to include only the depth image of pedestrian B as the depth image of the object of interest. Alternatively, the depth image of the standee containing only person A can be filtered to include only the depth image of the object of interest. Filtering the depth image of the object of interest from the depth image of the area to be analyzed can also be done by cropping the depth image of the area to be analyzed to obtain the depth image of the object of interest.

[0091] As mentioned above, the regional image of the area to be analyzed and the depth image of the area to be analyzed have a corresponding relationship. The image of the object of interest and the depth image of the object of interest also have a corresponding relationship. Naturally, based on the obtained image of the object of interest, the depth image of the area to be analyzed can be screened for the depth image that has this corresponding relationship with the image of the object of interest as the depth image of the object of interest.

[0092] Step S104: determining whether the object of interest is a first object of interest based on the object of interest image and the depth image of the object of interest.

[0093] After obtaining the attention object image and the depth image of the attention object, it may be determined whether the attention object is the first attention object based on the attention object image and the depth image of the attention object.

[0094] Specifically, as an implementation method of judging whether the object of interest is the first object of interest based on the image of the object of interest and the depth image of the object of interest: first, the image of the object of interest and the depth image of the object of interest are used as input data of the target convolutional neural network model to obtain attribute feature information of the object of interest. The target convolutional neural network model is a model for obtaining object attribute feature information of an image based on an image; then, based on the attribute feature information of the object of interest, it is judged whether the object of interest is the first object of interest.

[0095] The target convolutional neural network model is a trained feature extraction model that can extract attribute features of objects in an image. For example, the VGG16 network model is a relatively deep convolutional neural network model. The target convolutional neural network model is obtained by training the initial convolutional neural network model. The principles of the training process can be compared to the process of obtaining the image object detection model described above.

[0096] As a way to obtain the target convolutional neural network model, the initial convolutional neural network model can be trained. Specifically, the process of training the initial convolutional neural network model is as follows.

[0097] First, a second image sample is obtained for training the initial convolutional neural network model. Then, the initial convolutional neural network model is trained using the second image sample until the trained convolutional neural network model can accurately extract the attribute features of the object in the image.

[0098] Specifically, the above-mentioned second image samples can be images in a shopping mall scene and corresponding depth images, and the attribute characteristics of the objects in these second image samples are known. When the initial convolutional neural network model is trained, the second image samples are used as input data of the initial convolutional neural network model, and the attribute characteristics of the objects corresponding to the second image samples are used as output results to adjust the parameters in the initial convolutional neural network model. By using a large number of second image samples to train it and continuously adjusting the parameters in the initial convolutional neural network model, the target convolutional neural network model is finally obtained.

[0099] In this embodiment, attribute feature information of the object of interest is obtained based on the image of the object of interest and the depth image of the object of interest because the attribute feature information of different types of objects of interest obtained based on the depth image of the object of interest is significantly different. Therefore, based on the attribute feature information of the object of interest, it can be determined whether the object of interest is the first object of interest.

[0100] For example, pedestrians have a distinct human-like outline and 3D structure, while human-shaped standees are often flat and lack a human-like outline or 3D structure. For example, some human-shaped standees may be relatively regular circles or squares, with their edges not cut out to resemble a human outline. This human-shaped outline and 3D structure information is an example of attribute feature information. This human-shaped outline and 3D structure information can be obtained based on a depth image of the object of interest.

[0101] After obtaining the attribute feature information of the object of interest, it is determined whether the object of interest is the first object of interest based on the attribute feature information of the object of interest. In this embodiment, as an example of the first object of interest, a pedestrian is taken as the first object of interest.

[0102] Specifically, in order to understand how to determine whether the object of interest is the first object of interest based on the object of interest image and the depth image of the object of interest, please refer to Figure 3 ,exist Figure 3In the process, the cropped image (the cropped image includes the image of pedestrian B and the depth image of pedestrian B, or the cropped image includes the image of the human-shaped standee of person A and the depth image of the human-shaped standee containing person A) is input into the VGG16 network model to obtain its attribute feature information, and then classification and recognition are performed based on the attribute feature information, that is, distinguishing whether it is a real person or a "fake" person. In the cropped image, the depth image of the pedestrian is quite different from the depth image of the human-shaped standee, and thus the pedestrian can be accurately identified and distinguished based on its depth image.

[0103] Step S105: If the object of interest is the first object of interest, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined, where the avoidance strategy is used to control the self-propelled device to move.

[0104] In this embodiment, when it is determined that the object of interest is the first object of interest, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined.

[0105] In this embodiment, the first object of interest can be not only pedestrians but also other intelligent devices capable of actively avoiding autonomous vehicles. In this embodiment, pedestrians are primarily used as the first object of interest for illustration. Of course, it should be understood that the pedestrian in this embodiment does not simply refer to a person walking, but rather a person capable of autonomous walking and thus avoiding an autonomous vehicle.

[0106] Since the first object of interest is an object that the self-propelled device can actively avoid, after determining that the object of interest is the first object of interest, a mutual avoidance strategy between the self-propelled device and the first object of interest can be determined as follows: first, the self-propelled device is controlled to issue a voice reminder to avoid the first object of interest; then, if the first object of interest does not avoid the first object of interest within a preset time, the self-propelled device's travel path is replanned to avoid the first object of interest. The replanned path is used to control the movement of the self-propelled device.

[0107] The reason why voice reminders for the first object of concern are given priority to avoid is that such an avoidance strategy can prevent the self-propelled device from taking extra detours during the cleaning process. While avoiding wasting the battery of the self-propelled device, it can also avoid repeated cleaning of some areas, and not cleaning the area where the first object of concern is located. The most important thing is to protect the safety of pedestrians and self-propelled devices. For example, in a shopping mall scene, there may be some pedestrians walking or children running. In order to avoid collisions with pedestrians or children by self-propelled devices, the self-propelled device can be controlled to actively voice remind pedestrians or children to avoid or remind them to pay attention to safety when the self-propelled device is within a certain distance from the pedestrians or children.

[0108] Of course, if the first object of interest does not actively avoid the self-propelled device within the preset time, for example, a pedestrian sitting in a certain position or a child showing interest in the self-propelled device and not actively avoiding the self-propelled device for a period of time, the self-propelled device's route can be changed to improve the cleaning efficiency of the self-propelled device. For example, the route can be replanned to bypass the first object of interest.

[0109] In this embodiment, if the object of interest is determined to be a second object of interest, a movement control strategy is determined for the autonomous vehicle to avoid the second object of interest. The second object of interest may be an object that the autonomous vehicle cannot actively avoid, for example, the aforementioned human-shaped stand containing person A.

[0110] As a way to determine the movement control strategy for the self-propelled device to avoid the second object of interest: the travel path of the self-propelled device can be replanned to determine the replanned path; the replanned path is used to control the movement of the self-propelled device. For example, when the second object of interest is a human-shaped stand, the path for the self-propelled device can be replanned, and then the self-propelled device can be controlled to bypass the human-shaped stand to avoid knocking down the human-shaped stand. If the object of interest is the second object of interest, the travel path of the self-propelled device can be replanned to determine the replanned path when the distance between the self-propelled device and the second object of interest is less than a preset second distance threshold. The second distance threshold can be a preset more reasonable value, such as half a meter.

[0111] Of course, when other objects are identified in the area to be analyzed, you may also choose to lightly touch the object to clean it. For example, when there is a table in front of the self-propelled device, in order to clean it thoroughly, you can control the self-propelled device to lightly touch the table legs to clean the surrounding ground environment.

[0112] Of course, it is understandable that when determining the mutual avoidance strategy between the self-propelled device and the first object of interest, it is also possible to pre-judge whether the first object of interest is on the path of the self-propelled device. If so, the mutual avoidance strategy between the self-propelled device and the first object of interest is determined.

[0113] Furthermore, when determining the mutual avoidance strategy between the autonomous device and the first object of interest, it may also be determined whether the distance between the first object of interest and the autonomous device is less than a preset first distance threshold. If so, the mutual avoidance strategy between the autonomous device and the first object of interest is determined. To ensure that the autonomous device does not collide with pedestrians, the first distance threshold may be a preset, reasonable value, such as one meter.

[0114] Generally speaking, when it is confirmed that there are pedestrians in the path of the self-propelled device, voice reminders can be given at a longer distance to the pedestrians to avoid or pay attention to safety. When it is confirmed that there are objects in the path of the self-propelled device, the path of the self-propelled device can be replanned at a closer distance.

[0115] For example, if pedestrian B is in the direction of travel of the autonomous vehicle and a certain distance in front of it, a voice reminder can be issued to avoid collision. The purpose of issuing a voice reminder when the pedestrian is a certain distance away from the autonomous vehicle is to attract the pedestrian's attention. If the distance between the pedestrian and the autonomous vehicle exceeds a preset first distance threshold, the pedestrian may be too far away, and the voice reminder may disturb the pedestrian.

[0116] For another example, when a child C is in the direction of travel of the self-propelled device and a certain distance in front of the self-propelled device, in order to prevent the self-propelled device from hitting the child, a voice reminder can be given to the child C to pay attention to safety.

[0117] At the same time, when the distance between the first object of interest and the self-propelled device is less than a preset first distance threshold, the self-propelled device may be controlled to slow down or stop moving.

[0118] The present application provides a method for controlling the movement of a self-propelled device. In this method, a regional image and a depth image of a region to be analyzed are first obtained. Based on the regional image of the region to be analyzed, it is detected whether the region to be analyzed contains an object of interest. If the object of interest is contained, an image of the object of interest and a depth image of the object of interest are obtained based on the object of interest, the regional image, and the depth image of the region to be analyzed. Based on the image of the object of interest and the depth image of the object of interest, it is determined whether the object of interest is a first object of interest. If the object of interest is a first object of interest, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined, and the avoidance strategy is used to control the movement of the self-propelled device. In fact, after the method detects that the region to be analyzed contains an object of interest, based on the regional image, the image of the object of interest and the depth image of the object of interest are obtained based on the object of interest, the regional image, and the depth image of the region to be analyzed. Based on the depth image of the object of interest, it is possible to accurately determine whether the object of interest is the first object of interest. In this way, objects of similar types among the objects of interest can be accurately distinguished. When the object of interest is determined to be the first object of interest, a mutual avoidance strategy between the first object of interest and the self-propelled device is determined. Thus, while avoiding the self-propelled device from wasting energy by taking a detour, it is ensured that the self-propelled device and the first object of interest do not collide with each other, thereby ensuring the safety of the first object of interest.

[0119] Second embodiment

[0120] Corresponding to the self-propelled device movement control method provided in the first embodiment of this application, the second embodiment of this application also provides a self-propelled device movement control device. Since the device embodiment is basically similar to the first embodiment, the description is relatively simple. For relevant details, please refer to the partial description of the first embodiment. The device embodiment described below is merely illustrative.

[0121] Please refer to Figure 4 , which is a schematic diagram of the self-propelled equipment movement control device provided in the second embodiment of the present application.

[0122] The self-propelled device movement control device 400 includes:

[0123] The original image acquisition unit 401 is used to acquire a regional image and a depth image of the area to be analyzed;

[0124] The first judging unit 402 is configured to perform object detection on the region image and determine whether the detected objects include an object of interest;

[0125] The attention image obtaining unit 403 is configured to obtain an attention object image and a depth image of the attention object based on the attention object, the region image, and the depth image of the region to be analyzed if the detected objects include the attention object;

[0126] A second determining unit 404 is configured to determine whether the object of interest is a first object of interest based on the object of interest image and the depth image of the object of interest;

[0127] The mutual avoidance strategy determining unit 405 is configured to determine a mutual avoidance strategy between the self-propelled device and the first object of interest if the object of interest is the first object of interest, wherein the avoidance strategy is used to control the movement of the self-propelled device.

[0128] Optionally, the mutual avoidance strategy determination unit is specifically configured to:

[0129] Controlling the self-propelled device to give a voice reminder to avoid the first object of interest;

[0130] If the first object of interest is not avoided within a preset time, the travel path of the self-propelled device is replanned to avoid the first object of interest.

[0131] Optionally, the method further includes: a movement control strategy determining unit, configured to determine a movement control strategy for the self-propelled device to avoid the second object of interest if the object of interest is the second object of interest.

[0132] Optionally, the mutual avoidance strategy determination unit is specifically configured to:

[0133] It is determined whether the first object of interest is on a path of the self-propelled device. If so, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined.

[0134] Optionally, the mutual avoidance strategy determination unit is specifically configured to:

[0135] It is determined whether the distance between the first object of interest and the self-propelled device is less than a preset first distance threshold. If so, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined.

[0136] Optionally, the first judgment unit is specifically configured to:

[0137] Using the region image as input data of an image object detection model to obtain an object detection result of the region image, wherein the image object detection model is a model for obtaining an object detection result of an image based on an image;

[0138] Based on the object detection result of the area image, it is determined whether the detected objects include the object of interest.

[0139] Optionally, the focus image obtaining unit is specifically configured to:

[0140] According to the object of interest, the region image is cropped to obtain an image of the object of interest;

[0141] According to the object of interest, the depth image of the object of interest is filtered in the depth image of the area to be analyzed.

[0142] Optionally, the depth image of the area to be analyzed is acquired by using a depth sensor installed on the self-propelled device.

[0143] Optionally, the second judgment unit is specifically configured to:

[0144] Using the image of the object of interest and the depth image of the object of interest as input data of a target convolutional neural network model to obtain attribute feature information of the object of interest, wherein the target convolutional neural network model is a model for obtaining object attribute feature information of an image based on an image;

[0145] According to the attribute characteristic information of the object of interest, it is determined whether the object of interest is the first object of interest.

[0146] Optionally, the mobility control strategy determination unit is specifically configured to:

[0147] The travel path of the self-propelled device is replanned to determine a replanned path; the replanned path is used to control the self-propelled device to move.

[0148] Optionally, the avoidance strategy includes at least one of the following:

[0149] Prompt the first concerned object to avoid the target through voice prompting;

[0150] Controlling the self-propelled device to slow down or stop moving;

[0151] Changing the travel path of the self-propelled device.

[0152] Third embodiment

[0153] Corresponding to the method of the first embodiment of the present application, the third embodiment of the present application further provides an electronic device.

[0154] like Figure 5 As shown, Figure 5 This is a schematic diagram of an electronic device provided in the third embodiment of the present application.

[0155] In this embodiment, an optional hardware structure of the electronic device 500 can be as follows: Figure 5 As shown, it includes: at least one processor 501 , at least one memory 502 and at least one communication bus 505 ; the memory 502 contains a program 503 and data 504 .

[0156] The bus 505 can be a communication device for transmitting data between components inside the electronic device 500, such as an internal bus (for example, a CPU-memory bus, the central processing unit is a central processing unit, referred to as CPU), an external bus (for example, a universal serial bus port, a peripheral component interconnect express port), etc.

[0157] In addition, the electronic device further includes: at least one network interface 506 and at least one peripheral interface 507. The network interface 506 provides wired or wireless communication with an external network 508 (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.); in some embodiments, the network interface 506 may include any number of network interface controllers (NICs), radio frequency (RF) modules, repeaters, transceivers, modems, routers, gateways, any combination of wired network adapters, wireless network adapters, Bluetooth adapters, infrared adapters, near field communication (NFC) adapters, cellular network chips, etc.

[0158] The peripheral interface 507 is used to connect to the peripheral device, and the peripheral device can be the peripheral device 1 ( Figure 5 509 in), peripheral 2 ( Figure 5 510) and peripheral 3 ( Figure 5 511 in the figure). Peripherals are peripheral devices, which may include but are not limited to cursor control devices (such as a mouse, touchpad or touch screen), keyboards, displays (such as cathode ray tube displays, liquid crystal displays or light emitting diode displays), video input devices (such as cameras or input interfaces coupled to video files), etc.

[0159] The processor 501 may be a CPU, or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0160] The memory 502 may include a high-speed RAM (full name: Random Access Memory) memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0161] The processor 501 calls the program and data stored in the memory 502 to execute the method provided in the first embodiment of the present application.

[0162] Fourth embodiment

[0163] Corresponding to the method of the first embodiment of the present application, the fourth embodiment of the present application further provides a computer storage medium, which stores a computer program, and the computer program is run by a processor to execute the method provided by the first embodiment of the present application.

[0164] Fifth embodiment

[0165] Corresponding to the first embodiment, the fifth embodiment of the present application provides a self-propelled device, and the self-propelled device can refer to the relevant description in the first embodiment.

[0166] The self-propelled device includes: an image acquisition device, a depth sensor and a processor;

[0167] The image acquisition device is used to acquire a regional image of the area to be analyzed;

[0168] The depth sensor is used to obtain a depth image of the area to be analyzed;

[0169] The processor is used to receive a regional image of the area to be analyzed transmitted by the image acquisition device and a depth image of the area to be analyzed transmitted by the depth sensor, and perform object detection on the regional image to determine whether the detected objects include the object of interest; if the detected objects include the object of interest, then based on the object of interest, the regional image and the depth image of the area to be analyzed, obtain an image of the object of interest and a depth image of the object of interest; based on the image of the object of interest and the depth image of the object of interest, determine whether the object of interest is a first object of interest; if the object of interest is the first object of interest, determine a mutual avoidance strategy between the self-propelled device and the first object of interest, and the avoidance strategy is used to control the self-propelled device to move.

[0170] Application Scenario 1

[0171] When the cleaning robot is cleaning the floor in a shopping mall environment, it uses the RGBD camera installed in front of the cleaning robot to take real-time photos of the cleaning environment around it while cleaning, and then obtains a two-dimensional image of the cleaning environment and a depth image of the cleaning environment. The processor inside the cleaning robot judges based on the two-dimensional image of the cleaning environment and the depth image of the cleaning environment that there is a pedestrian walking towards the cleaning robot 0.75 meters in the cleaning direction directly in front of the cleaning robot. At this time, the processor inside the cleaning robot controls the cleaning robot to slow down and control the voice interaction device on the cleaning robot to give a voice reminder to the pedestrian, so that the pedestrian notices the cleaning robot and actively avoids the cleaning robot.

[0172] Application Scenario 2

[0173] When the cleaning robot is cleaning the floor, the processor inside the cleaning robot controls the voice interaction device on the cleaning robot to give voice reminders to avoid pedestrians. However, after a period of time, it detects that the pedestrian has not actively avoided the pedestrian. The processor inside the cleaning robot replans the cleaning robot's travel path to avoid the pedestrian, and at the same time controls the cleaning robot to continue cleaning the floor according to the replanned path.

[0174] Application Scenario 3

[0175] When the cleaning robot is cleaning the floor in a shopping mall environment, it uses the RGBD camera installed in front of the cleaning robot to take real-time photos of the cleaning environment around it while cleaning, and then obtains a two-dimensional image and a depth image of the cleaning environment. Based on the two-dimensional image and the depth image of the cleaning environment, the processor inside the cleaning robot determines that there is a human-shaped stand 0.3 meters in the cleaning direction directly in front of the cleaning robot. At this time, the processor inside the cleaning robot re-plans the cleaning robot's travel path to avoid the human-shaped stand, and at the same time controls the cleaning robot to continue cleaning the floor according to the re-planned path.

[0176] Although the present application is disclosed as above with the preferred embodiments, it is not intended to limit the present application. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0177] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in computer-readable media, random access memory (RAM), and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0178] 1. Computer-readable media, including permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for information storage. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable storage media, such as modulated data signals and carrier waves.

[0179] 2. Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A method for controlling movement of a self-propelled device, characterized in that: include: Acquire a regional image and a depth image of the area to be analyzed; Performing object detection on the regional image to determine whether the detected objects include the object of interest; If the detected objects include an object of interest, obtaining an image of the object of interest and a depth image of the object of interest based on the object of interest, the area image, and the depth image of the area to be analyzed; determining, based on the object of interest image and the depth image of the object of interest, whether the object of interest is a first object of interest; If the object of interest is a first object of interest, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined, where the avoidance strategy is used to control the self-propelled device to move.

2. The method for controlling movement of a self-propelled device according to claim 1, wherein: The determining of a mutual avoidance strategy between the self-propelled device and the first object of interest includes: Controlling the self-propelled device to give a voice reminder to avoid the first object of interest; If the first object of interest is not avoided within a preset time, the travel path of the self-propelled device is replanned to avoid the first object of interest.

3. The method for controlling movement of a self-propelled device according to claim 1, wherein: Also includes: If the object of interest is a second object of interest, a movement control strategy for the self-propelled device to avoid the second object of interest is determined.

4. The method for controlling movement of a self-propelled device according to claim 1, wherein: The determining of a mutual avoidance strategy between the self-propelled device and the first object of interest includes: It is determined whether the first object of interest is on a path of the self-propelled device. If so, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined.

5. The method for controlling movement of a self-propelled device according to claim 1, wherein: The determining of a mutual avoidance strategy between the self-propelled device and the first object of interest includes: It is determined whether the distance between the first object of interest and the self-propelled device is less than a preset first distance threshold. If so, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined.

6. The method for controlling movement of a self-propelled device according to claim 1, wherein: The performing object detection on the regional image and determining whether the detected objects include the object of interest includes: Using the region image as input data of an image object detection model to obtain an object detection result of the region image, wherein the image object detection model is a model for obtaining an object detection result of an image based on an image; Based on the object detection result of the area image, it is determined whether the detected objects include the object of interest.

7. The method for controlling movement of a self-propelled device according to claim 1, wherein: The obtaining of an image of the object of interest and a depth image of the object of interest based on the object of interest, the region image, and the depth image of the area to be analyzed includes: According to the object of interest, the region image is cropped to obtain an image of the object of interest; According to the object of interest, the depth image of the object of interest is filtered in the depth image of the area to be analyzed.

8. The method for controlling movement of a self-propelled device according to claim 7, wherein: The depth image of the area to be analyzed is obtained by using a depth sensor installed on the self-propelled device.

9. The method for controlling movement of a self-propelled device according to claim 1, wherein: The determining whether the object of interest is a first object of interest based on the object of interest image and the depth image of the object of interest includes: Using the image of the object of interest and the depth image of the object of interest as input data of a target convolutional neural network model to obtain attribute feature information of the object of interest, wherein the target convolutional neural network model is a model for obtaining object attribute feature information of an image based on an image; According to the attribute characteristic information of the object of interest, it is determined whether the object of interest is the first object of interest.

10. The method for controlling movement of a self-propelled device according to claim 3, wherein: The determining of a movement control strategy for the self-propelled device to avoid the second object of interest includes: The travel path of the self-propelled device is replanned to determine a replanned path; the replanned path is used to control the self-propelled device to move.

11. The method for controlling movement of a self-propelled device according to claim 1, wherein: The avoidance strategy includes at least one of the following: Prompt the first concerned object to avoid the target through voice prompting; Controlling the self-propelled device to slow down or stop moving; Changing the travel path of the self-propelled device.

12. A self-propelled equipment movement control device, characterized in that: include: An original image acquisition unit, configured to acquire a regional image and a depth image of the region to be analyzed; a first judging unit, configured to perform object detection on the region image and judge whether the detected objects include an object of interest; an image of interest obtaining unit, configured to obtain an image of the object of interest and a depth image of the object of interest based on the object of interest, the region image, and the depth image of the region to be analyzed, if the detected objects include the object of interest; a second determining unit, configured to determine whether the object of interest is a first object of interest based on the object of interest image and the depth image of the object of interest; The mutual avoidance strategy determining unit is configured to determine a mutual avoidance strategy between the self-propelled device and the first object of interest if the object of interest is the first object of interest, wherein the avoidance strategy is used to control the self-propelled device to move.

13. An electronic device, characterized in that: include: processor; A memory for storing a computer program, wherein the computer program is executed by a processor to perform the method according to any one of claims 1 to 11.

14. A computer storage medium, characterized in that The computer storage medium stores a computer program, which is executed by a processor to perform the method according to any one of claims 1 to 11.

15. A self-propelled device, characterized in that: include: Image acquisition devices, depth sensors and processors; The image acquisition device is used to acquire a regional image of the area to be analyzed; The depth sensor is used to obtain a depth image of the area to be analyzed; The processor is configured to receive a regional image of the area to be analyzed transmitted by the image acquisition device and a depth image of the area to be analyzed transmitted by the depth sensor, and perform object detection on the regional image to determine whether the detected objects include an object of interest; If the detected objects include an object of interest, obtaining an image of the object of interest and a depth image of the object of interest based on the object of interest, the area image, and the depth image of the area to be analyzed; determining, based on the object of interest image and the depth image of the object of interest, whether the object of interest is a first object of interest; If the object of interest is a first object of interest, a mutual avoidance strategy between the self-propelled device and the first object of interest is determined, where the avoidance strategy is used to control the self-propelled device to move.

Images