Object information determination method, mobile robot system, and electronic device

Abstract

Embodiments of the present application provide an object information determination method, a mobile robot system and an electronic device, and are applied to the technical field of robots. The method comprises: acquiring a plurality of images containing a target object collected by a mobile robot in a process of performing a surrounding motion on the target object; for each acquired image, identifying a relative distance between the mobile robot and the target object when the image is collected; and determining contour information of the target object based on the relative distance between the mobile robot and the target object when each image is collected and position information of the mobile robot in a world coordinate system. Through the present solution, the contour information of an object in an environment can be determined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mobile robots, and in particular to an object information determination method, a mobile robot system and an electronic device. BACKGROUND

[0002] In recent years, with the continuous development of robot technology, mobile robots play an increasingly important role in life, such as household or commercial sweeping robots, welcome robots, etc.

[0003] In order to avoid colliding with objects during movement, obstacle avoidance function is a basic function required to be implemented by mobile robots, and the implementation of obstacle avoidance function requires that the mobile robot can perceive the contour information of objects in the environment. Therefore, how to determine the contour information of objects in the environment is a technical problem to be solved. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide an object information determination method, a mobile robot system and an electronic device to determine the contour information of objects in the environment. The specific technical solutions are as follows:

[0005] In a first aspect, the embodiments of the present application provide an object information determination method, which comprises:

[0006] acquiring a plurality of images containing a target object collected by a mobile robot during a surrounding motion of the mobile robot with respect to the target object;

[0007] for each acquired image, identifying the relative distance between the mobile robot and the target object when the mobile robot collects the image;

[0008] based on the relative distance between the mobile robot and the target object when the mobile robot collects each image and the position information in the world coordinate system, determining the contour information of the target object.

[0009] Optionally, the step of identifying the relative distance between the mobile robot and the target object when the mobile robot collects each acquired image comprises:

[0010] for each acquired image, determining the pixel distance of a first pixel point and a second pixel point in the image in the vertical direction as a first distance; wherein the first pixel point is a bottom pixel point of the target object, and the second pixel point is a center pixel point;

[0011] based on the first distance and the first internal and external parameter information of the camera in the mobile robot, determining the relative distance between the mobile robot and the target object when the mobile robot collects the image.

[0012] Optionally, the first internal and external reference information comprises a vertical angle of view of the camera, an image size of the image captured by the camera, and a height of an optical center of the camera.

[0013] The first distance and first internal and external reference information of the camera of the mobile robot are used to determine a relative distance between the mobile robot and the target object when the frame of image is captured.

[0014] A lower view line angle is determined based on the first distance, a vertical resolution of the frame of image, and a vertical angle of view of the camera. The lower view line angle is an angle between a lower view line and an optical axis of the camera. The lower view line is a line between an optical center of the camera and a bottom of the target object.

[0015] The relative distance between the mobile robot and the target object when the frame of image is captured is calculated based on the lower view line angle and a height of the optical center of the camera.

[0016] Optionally, the lower view line angle is determined based on the first distance, the vertical resolution of the frame of image, and the vertical angle of view of the camera.

[0017] A first ratio of the first distance to the vertical resolution is calculated as a first ratio.

[0018] The lower view line angle is determined based on the first ratio and the vertical angle of view.

[0019] Optionally, the lower view line angle is determined based on the first ratio and the vertical angle of view.

[0020] A product of the first ratio and the vertical angle of view is calculated as the lower view line angle, or

[0021] A target adjustment coefficient corresponding to a specified pixel point in the frame of image is determined according to a preset corresponding relationship between each pixel point and adjustment coefficient. A product of the adjustment coefficient and the first ratio is calculated as a second ratio, and a product of the second ratio and the vertical angle of view is calculated as the lower view line angle. The each pixel point is a pixel point in the image captured by the camera, and the specified pixel point is a pixel point determined from bottom pixel points of the target object.

[0022] Optionally, the relative distance between the mobile robot and the target object when the frame of image is captured is calculated based on the lower view line angle and the height of the optical center of the camera.

[0023] The relative distance between the mobile robot and the target object when the frame of image is captured is calculated by using the following formula:

[0024] d = tan (90 + k - m) * h

[0025] wherein, d is the relative distance, k is the vertical angle between the optical axis and the moving plane of the mobile robot, m is the downward line angle, and h is the optical center height of the camera.

[0026] Optionally, the determining the contour information of the target object based on the relative distance between the mobile robot and the target object when each frame of image is collected and the position information of the target object in the world coordinate system comprises:

[0027] For each frame of image, the position of a target edge point of the target object is determined based on the position information of the mobile robot in the world coordinate system when the frame of image is collected and the relative distance between the mobile robot and the target object, as the edge position corresponding to the frame of image; wherein the target edge point is an edge point of the camera in the shooting range when the frame of image is collected.

[0028] The contour information of the target object is determined based on the edge positions corresponding to each frame of image.

[0029] Optionally, the determining the contour information of the target object based on the edge positions corresponding to each frame of image comprises:

[0030] the edge positions corresponding to each frame of image are taken as the contour information of the target object; or,

[0031] curve fitting is performed on the edge positions corresponding to each frame of image to obtain at least one fitting curve, and the positions of points on the at least one fitting curve are determined, and the determined positions are taken as the contour information of the target object.

[0032] Optionally, before the step of obtaining the plurality of frames of image containing the target object collected in the process of the mobile robot performing the surrounding motion for the target object, the method further comprises:

[0033] performing object type identification on the target object;

[0034] if the object type of the target object is a to-be-identified type, performing the step of obtaining the plurality of frames of image containing the target object collected in the process of the mobile robot performing the surrounding motion for the target object; wherein the to-be-identified type comprises an unknown object type or a non-fixed shape object type.

[0035] Optionally, the method further comprises:

[0036] If the object type of the target object is a fixed object type, determine preset initial information corresponding to the object type of the target object as to-be-utilized information; the initial information indicates initial positions of the edge points.

[0037] Obtain an image collected by the mobile robot for the target object, and identify a relative distance between the mobile robot and the target object when the image is collected.

[0038] Based on the identified relative position, adjust the initial positions of the edge points indicated by the to-be-utilized information to obtain adjusted positions of the edge points as contour information of the target object.

[0039] Optionally, the manner in which the mobile robot moves around the target object comprises:

[0040] In a case where the orientation of the camera of the mobile robot is irrelevant to the moving direction of the mobile robot, the mobile robot performs continuous circular motion around the target object; or,

[0041] In a case where the orientation of the camera of the mobile robot is relevant to the moving direction of the mobile robot, the mobile robot performs multi-segment arc motion around the target object.

[0042] Optionally, after the relative distance between the mobile robot and the target object when the frame image is collected is identified, the method further comprises:

[0043] Based on the relative distance between the mobile robot and the target object when the frame image is collected, a second distance, and second internal and external parameter information of the camera, determine the height of the target object.

[0044] The second distance is a pixel distance in a vertical direction between a top pixel point of the target object in the frame image and a center pixel point of the frame image.

[0045] Optionally, the second internal and external parameter information comprises a vertical viewing angle of the camera and a vertical resolution of the image collected by the camera.

[0046] The determining of the height of the target object based on the relative distance between the mobile robot and the target object when the frame image is collected, the second distance, and the second internal and external parameter information of the camera comprises:

[0047] The height of the target object is determined by using the following formula:

[0048]

[0049] Wherein, x is the height of the target object, θ is the vertical angle of view, dv2_pixels is the second distance, V is the vertical resolution, and d is the relative distance between the mobile robot and the target object when the mobile robot collects the image.

[0050] Optionally, after the mobile robot performs the surrounding motion for the target object, the method further comprises:

[0051] For each acquired image, the processor identifies a horizontal deflection angle between the mobile robot and the target object when the mobile robot collects the image; wherein the horizontal deflection angle is a horizontal angle between a horizontal line of sight and an optical axis of a camera in the mobile robot, and the horizontal line of sight is a line between an optical center of the camera and an outside of the target object.

[0052] In a second aspect, an embodiment of the present application provides a mobile robot system, comprising:

[0053] An image acquisition module is configured to acquire a plurality of images containing a target object, which are collected by a mobile robot during a surrounding motion of the mobile robot for the target object.

[0054] A processor is configured to, for each acquired image, identify a relative distance between the mobile robot and the target object when the mobile robot collects the image; and determine contour information of the target object based on the relative distances between the mobile robot and the target object when the mobile robot collects the images and position information of the mobile robot in a world coordinate system.

[0055] Optionally, the mobile robot system further comprises:

[0056] A power module is configured to drive the mobile robot to perform the surrounding motion for the target object.

[0057] In a third aspect, an embodiment of the present application provides an object information determination apparatus, comprising:

[0058] An image acquisition module is configured to acquire a plurality of images containing a target object, which are collected by a mobile robot during a surrounding motion of the mobile robot for the target object.

[0059] An information calculation module is configured to, for each acquired image, identify a relative distance between the mobile robot and the target object when the mobile robot collects the image.

[0060] An information determination module is configured to determine contour information of the target object based on the relative distances between the mobile robot and the target object when the mobile robot collects the images and position information of the mobile robot in a world coordinate system.

[0061] Optionally, the information calculation module comprises:

[0062] a first sub-module configured to determine, for each acquired image frame, a pixel distance between a first pixel point and a second pixel point in the frame image in a vertical direction as a first distance, wherein the first pixel point is a bottom pixel point of the target object and the second pixel point is a center pixel point;

[0063] a second sub-module configured to determine, based on the first distance and first internal and external parameter information of a camera in the mobile robot, a relative distance between the mobile robot and the target object when the frame image is acquired.

[0064] Optionally, the first internal and external parameter information comprises a vertical viewing angle of the camera, an image size of an image acquired by the camera, and a light center height of the camera.

[0065] The second sub-module comprises:

[0066] an included angle determination unit configured to determine, based on the first distance, a vertical resolution of the frame image, and the vertical viewing angle of the camera, a downward viewing line included angle, wherein the downward viewing line included angle is an included angle between a downward viewing line and an optical axis of the camera, and the downward viewing line is a line connecting a light center of the camera and a bottom of the target object.

[0067] a distance determination unit configured to calculate, based on the downward viewing line included angle and the light center height of the camera, the relative distance between the mobile robot and the target object when the frame image is acquired.

[0068] Optionally, the included angle determination unit comprises:

[0069] a ratio calculation sub-unit configured to calculate a ratio of the first distance to the vertical resolution as a first ratio.

[0070] an included angle determination sub-unit configured to determine, based on the first ratio and the vertical viewing angle, the downward viewing line included angle.

[0071] Optionally, the included angle determination sub-unit is specifically configured to calculate a product of the first ratio and the vertical viewing angle as the downward viewing line included angle, or to determine, according to a preset corresponding relationship between each pixel point and an adjustment coefficient, a target adjustment coefficient corresponding to a specified pixel point in the frame image, calculate a product of the adjustment coefficient and the first ratio as a second ratio, and calculate a product of the second ratio and the vertical viewing angle as the downward viewing line included angle, wherein the each pixel point is a pixel point in an image acquired by the camera, and the specified pixel point is a pixel point determined from the bottom pixel point of the target object.

[0072] Optionally, the distance determining unit is specifically configured to calculate the relative distance between the mobile robot and the target object when the mobile robot collects the frame of image by using the following formula:

[0073] d=tan(90+k-m)*h

[0074] wherein d is the relative distance, k is a vertical angle between the optical axis and a moving plane of the mobile robot, m is the downward viewing angle, and h is a height of an optical center of the camera.

[0075] Optionally, the information determining module comprises:

[0076] a position determining sub-module configured to determine, for each frame of image, a position of a target edge point of the target object based on position information of the mobile robot in a world coordinate system when the mobile robot collects the frame of image and the relative distance between the mobile robot and the target object, as an edge position corresponding to the frame of image, wherein the target edge point is an edge point of a shooting range of the camera when the mobile robot collects the frame of image.

[0077] an information determining sub-module configured to determine contour information of the target object based on the edge positions corresponding to the frames of image.

[0078] Optionally, the information determining sub-module is specifically configured to take the edge positions corresponding to the frames of image as the contour information of the target object, or perform curve fitting on the edge positions corresponding to the frames of image to obtain at least one fitting curve and determine positions of points on the at least one fitting curve, and take the determined positions as the contour information of the target object.

[0079] Optionally, the apparatus further comprises a type identifying module configured to identify an object type of the target object before the image obtaining module performs the step of obtaining the multiple frames of image containing the target object collected by the mobile robot during the process of the surrounding motion of the target object, and call the image obtaining module to perform the step of obtaining the multiple frames of image containing the target object collected by the mobile robot during the process of the surrounding motion of the target object if the object type of the target object is a to-be-identified type, wherein the to-be-identified type comprises an unknown object type or a non-fixed shape object type.

[0080] Optionally, the type identifying module is further configured to: if the object type of the target object is a fixed object type, determine preset initial information corresponding to the object type of the target object as the to-be-utilized information; acquire an image of the target object collected by the mobile robot, and identify a relative distance between the mobile robot and the target object when the image is collected; and based on the identified relative distance, adjust initial positions of edge points indicated by the to-be-utilized information to obtain adjusted positions of the edge points as the contour information of the target object; wherein the initial information indicates the initial positions of the edge points.

[0081] Optionally, the manner in which the mobile robot performs the surrounding motion for the target object comprises: in a case where a camera of the mobile robot is oriented independently of a moving direction of the mobile robot, the mobile robot performs continuous circular motion for the target object; or in a case where the camera of the mobile robot is oriented in relation to the moving direction of the mobile robot, the mobile robot performs multi-segment arc motion for the target object.

[0082] Optionally, the information calculating module is further configured to, after identifying the relative distance between the mobile robot and the target object when the frame of image is collected, determine a height of the target object based on the relative distance between the mobile robot and the target object when the frame of image is collected, a second distance, and second internal and external parameter information of the camera; wherein the second distance is a pixel distance in a vertical direction between a top pixel point of the target object in the frame of image and a center pixel point of the frame of image.

[0083] Optionally, the second internal and external parameter information comprises a vertical viewing angle of the camera and a vertical resolution at which the camera collects images.

[0084] The information calculating module comprises:

[0085] The height calculating sub-module is configured to determine the height of the target object by using the following formula:

[0086]

[0087] wherein x is the height of the target object, θ is the vertical viewing angle, dv2_pixels is the second distance, V is the vertical resolution, and d is the relative distance between the mobile robot and the target object when the frame of image is collected.

[0088] Optionally, an angle identification module is configured to identify, for each frame of image, a horizontal deflection angle of the mobile robot relative to the target object when the mobile robot collects the frame of image, after the image acquisition module collects a plurality of frames of image containing the target object during the process that the mobile robot performs the surrounding motion relative to the target object; wherein the horizontal deflection angle is a horizontal included angle between a horizontal line of sight and an optical axis of a camera in the mobile robot, and the horizontal line of sight is a line between an optical center of the camera and an outer side of the target object.

[0089] Optionally, the angle identification module comprises:

[0090] a distance determination sub-module configured to determine a pixel distance between a third pixel point and a fourth pixel point in the frame of image in a horizontal direction as a third distance, wherein the third pixel point is an outer pixel point relative to the target object, and the fourth pixel point is a center pixel point;

[0091] an angle determination sub-module configured to determine, based on the third distance, a horizontal resolution of the frame of image and a horizontal view angle of the camera, the horizontal deflection angle of the mobile robot relative to the target object when the mobile robot collects the frame of image.

[0092] Optionally, the angle determination sub-module is specifically configured to calculate a ratio of the third distance to the horizontal resolution as a third ratio, and determine, based on the third ratio and the horizontal view angle, the horizontal deflection angle of the mobile robot relative to the target object when the mobile robot collects the frame of image.

[0093] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus.

[0094] The memory is configured to store a computer program.

[0095] The processor is configured to execute the program stored on the memory to implement the method steps of any one of the first aspect.

[0096] In a fifth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method steps of any one of the first aspect.

[0097] The embodiment of the present application has the following beneficial effects:

[0098] In the object information determination method provided by the embodiment of the present application, the mobile robot can collect multiple images containing the target object in the process of performing the surrounding motion on the target object; for each collected image, the relative distance between the mobile robot and the target object when the mobile robot collects the image is identified; and based on the relative distance between the mobile robot and the target object when the mobile robot collects each image and the position information of the mobile robot in the world coordinate system, the contour information of the target object is determined. Since the multiple images are collected by the mobile robot in the process of performing the surrounding motion on the target object, the multiple images are images of the target object in different directions collected by the mobile robot, and the relative distance between the mobile robot and the target object when the mobile robot collects each image is the distance between the mobile robot and the contour edge of the target object when the mobile robot collects the image, and then the contour information of the target object can be determined in combination with the position information of the mobile robot in the world coordinate system when the mobile robot collects the image. It can be seen that the contour information of the object in the environment can be determined by the present solution.

[0099] Of course, implementing any product or method of the present application does not necessarily require achieving all the advantages described above at the same time. BRIEF DESCRIPTION OF DRAWINGS

[0100] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other embodiments can also be obtained from these drawings without creative labor.

[0101] Figure 1 The flowchart of the object information determination method provided by the embodiment of the present application;

[0102] FIG. 2(a) is a schematic diagram of a circular motion provided by the embodiment of the present application;

[0103] FIG. 2(b) is a schematic diagram of a multi-segment arc motion provided by the embodiment of the present application;

[0104] Figure 3 Another flowchart of the object information determination method provided by the embodiment of the present application;

[0105] Figure 4 A schematic diagram of an image containing a target object provided by the embodiment of the present application;

[0106] Figure 5 Another flowchart of the object information determination method provided by the embodiment of the present application;

[0107] Figure 6A schematic diagram of a global side view provided by an embodiment of the present application;

[0108] Figure 7 Another flowchart of the object information determination method provided by an embodiment of the present application;

[0109] Figure 8 Another schematic diagram of a global side view provided by an embodiment of the present application;

[0110] Figure 9 A structural schematic diagram of a mobile robot system provided by an embodiment of the present application;

[0111] Figure 10 Another structural schematic diagram of a mobile robot system provided by an embodiment of the present application;

[0112] Figure 11 A structural schematic diagram of an object information determination apparatus provided by an embodiment of the present application;

[0113] Figure 12 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0114] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0115] The obstacle avoidance function is a basic function required to be implemented by a mobile robot, and the implementation of the obstacle avoidance function requires that the mobile robot can perceive the contour information of objects in the environment. Therefore, how to determine the contour information of objects in the environment is a technical problem to be solved urgently.

[0116] In order to determine the contour information of objects in the environment, an embodiment of the present application provides an object information determination method, a mobile robot system and an electronic device.

[0117] It should be noted that in specific applications, the object information determination method provided by the embodiment of the present application can be applied to a mobile robot, such as a sweeping robot or a welcome robot. Alternatively, the object information determination method provided by the embodiment of the present application can also be applied to other various electronic devices, such as a smart phone, a personal computer, a server, and other devices with data processing capabilities. When applied to other various electronic devices, the electronic book can communicate with the mobile robot, so that the required image for processing can be obtained from the mobile robot. In addition, it can be understood that the object information determination method provided by the embodiment of the present application can be realized by software, hardware or a combination of software and hardware.

[0118] The object information determination method provided by the embodiment of the present application can include:

[0119] acquiring a plurality of images containing the target object collected by the mobile robot in the process of surrounding motion for the target object;

[0120] For each acquired image, identifying the relative distance between the mobile robot and the target object when collecting the image;

[0121] Based on the relative distance between the mobile robot and the target object when collecting each image and the position information in the world coordinate system, the contour information of the target object is determined.

[0122] In the above scheme provided by the embodiment of the present application, since the plurality of images acquired are collected by the mobile robot in the process of surrounding motion for the target object, the plurality of images acquired are images of different orientations of the target object collected by the mobile robot. The relative distance between the mobile robot and the target object when collecting each image is the distance between the mobile robot and the contour edge of the target object when collecting the image. Then, the contour information of the target object can be determined in combination with the position information of the mobile robot in the world coordinate system when collecting the image. It can be seen that the contour information of the object in the environment can be determined by the present scheme.

[0123] The object information determination method provided by the embodiment of the present application will be described in detail in conjunction with the accompanying drawings of the specification.

[0124] As shown in Figure 1 The object information determination method provided by the embodiment of the present application can include steps S101-S103, wherein:

[0125] S101, acquiring a plurality of images containing the target object collected by the mobile robot in the process of surrounding motion for the target object;

[0126] The target object can be an object that has not been obtained object information by the mobile robot during movement, for example, when the mobile robot is a sweeping robot, when the sweeping robot detects that the moving path is blocked by obstacles such as garbage cans, toys, stools, or slippers during movement, the obstacles that hinder the sweeping robot from moving according to the original moving path can be considered as the target object of the present application. Of course, the target object can also be an artificially designated object, which is acceptable.

[0127] The above-mentioned circumferential movement refers to a movement mode in which the mobile robot moves at least one circle around the target object, and the above-mentioned obtained multiple images can be images obtained by the mobile robot shooting the target object during circumferential movement.

[0128] In order to collect multiple images containing the target object during circumferential movement of the target object, the mobile robot can adopt various movement modes to achieve circumferential movement of the target object.

[0129] Optionally, the mobile robot can perform continuous circular movement on the target object.

[0130] The continuous circular movement refers to uninterrupted movement around the target object, as shown in FIG. 2(a), which is a schematic diagram of circular movement provided by an embodiment of the present application, the irregular body is the target object, and the direction indicated by the circular arc indicates that the mobile robot C performs counterclockwise circular movement on the target object. During the circular movement, the mobile robot C collects images of the target object at different positions of the target object.

[0131] Since the mobile robot needs to collect images of the target object during circular movement, the camera direction of the mobile robot is constantly changing with the movement direction of the mobile robot, so that the camera direction of the mobile robot is independent of the movement direction of the mobile robot when the mobile robot adopts circular movement, that is, the mobile robot can perform continuous circular movement on the target object when the camera direction of the mobile robot is independent of the movement direction of the mobile robot.

[0132] If the camera direction of the mobile robot is related to the movement direction of the mobile robot, for example, the camera direction of the mobile robot is the front direction of the mobile robot, at this time, if the mobile robot still adopts circular movement, it will cause the mobile robot to be unable to collect images of the target object during movement. In order to solve this problem, the present application further provides a movement mode of multi-segment arc movement.

[0133] As shown in FIG. 2(b), the embodiment of the present application provides a schematic diagram of multi-segment arc-shaped motion. The mobile robot can move around the target robot for a segment of circular arc, then adjust the camera orientation to face the target object, and collect an image of the target object, and then continue to move the next segment of circular arc, repeat the above process, until the target object is moved at least one circle.

[0134] It can be seen that, in the case that the camera orientation of the mobile robot is related to the moving direction of the mobile robot, the mobile robot performs multi-segment arc-shaped motion for the target object. Of course, in the case that the camera orientation of the mobile robot is irrelevant to the moving direction of the mobile robot, the multi-segment arc-shaped motion can also be used to complete the surrounding motion of the target object.

[0135] S102, for each frame of the acquired image, identify the relative distance between the mobile robot and the target object when collecting the frame of image;

[0136] For example, as shown in FIG. 2(a), when the image acquired by the mobile robot is collected when the mobile robot moves to the target object directly below, the relative distance between the mobile robot and the target object when collecting the frame of image can be determined by using the acquired image, which is distance d1.

[0137] When the mobile robot includes a depth camera, such as a structured light depth camera, a binocular camera, a TOF (Time of Flight) depth camera, and a binocular stereo vision camera, etc., the acquired image includes depth information, so that for each frame of the acquired image, the relative distance between the mobile robot and the target object when collecting the frame of image can be identified by using the depth information recorded in the image.

[0138] Of course, the hardware cost of the depth camera is high, in order to save the hardware cost, the embodiment of the present application also provides a way of identifying the relative distance between the mobile robot and the target object when collecting the frame of image based on the monocular camera collecting the image, which will be described in detail in subsequent embodiments, and will not be repeated here.

[0139] S103, based on the relative distance between the mobile robot and the target object when collecting each frame of image and the position information in the world coordinate system, determine the contour information of the target object.

[0140] After identifying the relative distance between the mobile robot and the target object when collecting each frame of image, the contour information of the target object can be determined based on the relative distance between the mobile robot and the target object when collecting each frame of image and the position information in the world coordinate system.

[0141] The position information of the mobile robot in the world coordinate system can be a three-dimensional coordinate of the mobile robot in the world coordinate system. The world coordinate system is a coordinate system established by the mobile robot when it is in a new environment. In general, the initial moving point (for example, the charging position of the robot) of the mobile robot is taken as the coordinate origin. During the movement of the mobile robot, the position of the mobile robot in the world coordinate system can be updated in real time in combination with the distance, direction and other information during the movement of the mobile robot.

[0142] In the embodiment of the application, the mobile robot can record the position information of the mobile robot in the world coordinate system when collecting each frame of image. If the execution subject of the embodiment of the application is an electronic device in communication with the mobile robot, the electronic device can obtain the position information of the mobile robot in the world coordinate system when the mobile robot collects each frame of image at the same time as the mobile robot collects the image.

[0143] Optionally, the position of the target edge point of the target object can be determined based on the position information of the mobile robot in the world coordinate system when collecting each frame of image and the relative distance from the target object, as the edge position corresponding to the frame of image, and then the contour information of the target object can be determined based on the edge positions corresponding to the frames of image.

[0144] The target edge point is an edge point of the camera in the shooting range of the mobile robot when collecting the frame of image. For example, in FIG. 2(a), when the mobile robot is located directly below the target object, the determined distance d1 is the distance between the mobile robot and the edge point directly below the target object, and the edge point is the edge point of the camera in the shooting range of the mobile robot at this time.

[0145] After the target edge point of each target object is determined, the contour information of the target object can be determined based on the edge positions corresponding to the frames of image.

[0146] The contour information of the target object can be determined in various ways based on the edge positions corresponding to the frames of image. Optionally, at least two contour information determination methods are included, which are as follows.

[0147] The first contour information determination method is to take the edge positions corresponding to the frames of image as the contour information of the target object.

[0148] In this mode, the edge positions corresponding to each frame of image can be directly taken as the contour information of the target object. The contour information of the target object can be a set of edge positions, and each determined edge position can be taken as an element in the set of edge positions. For example, the edge positions corresponding to each frame of image are position 1, position 2, position 3, position 4, position 5 and position 6 respectively, and the contour information of the target object is {position 1, position 2, position 3, position 4, position 5, position 6}.

[0149] The second contour information determination mode is as follows: curve fitting is performed on the edge positions corresponding to each frame of image to obtain at least one fitting curve, and the positions of each point on the at least one fitting curve are determined, and the determined positions are taken as the contour information of the target object.

[0150] Optionally, the curve fitting mode can be least square curve fitting, RBF (Radial Basis Function) curve fitting, cubic spline curve fitting or the like. After the at least one fitting curve is obtained, the positions of each point on the at least one fitting curve can be taken as the contour information of the target object.

[0151] In the above scheme provided by the embodiment of the application, the obtained multiple frames of image are collected by the mobile robot in the process of surrounding the target object, so that the obtained multiple frames of image are images of different orientations of the target object collected by the mobile robot. The relative distance between the mobile robot and the target object when each frame of image is collected is the distance between the mobile robot and the contour edge of the target object when the frame of image is collected. Then, the contour information of the target object can be determined in combination with the position information of the mobile robot in the world coordinate system when the frame of image is collected. It can be seen that the contour information of the object in the environment can be determined by the scheme.

[0152] In Figure 1 the embodiment shown, based on the embodiment shown in Figure 3 , the object information determination method provided by another embodiment of the application can include S102A-S102B, wherein:

[0153] S102A, for each frame of obtained image, determining the pixel distance in the vertical direction between the first pixel point and the second pixel point in the frame of image as the first distance; wherein the first pixel point is the bottom pixel point of the target object, and the second pixel point is the center pixel point.

[0154] As Figure 4 shown, the embodiment of the application provides a schematic diagram of an image containing a target object. In the diagram, the first pixel point can be any pixel point on the bottom line segment of the target object, and the second pixel point is the center pixel point in the diagram.

[0155] The first pixel point can be selected from the bottom line segment of the target object, and the first pixel coordinate of the first pixel point can be determined, and then the second pixel coordinate of the second pixel point can be determined. Then, the difference between the vertical coordinate in the first pixel coordinate and the vertical coordinate in the second pixel coordinate can be calculated, and the absolute value of the difference can be taken as the first distance. For example, the first pixel coordinate is (x1, y1), and the second pixel coordinate is (x2, y2). The first distance is the absolute value of y1-y2.

[0156] S102B, based on the first distance and the first internal and external parameter information of the camera in the mobile robot, determining the relative distance between the mobile robot and the target object when the mobile robot collects the frame image.

[0157] After determining the first distance, the relative distance between the mobile robot and the target object when the mobile robot collects the frame image can be determined based on the first distance and the first internal and external parameter information of the camera in the mobile robot according to the camera imaging principle and the geometric relationship between the camera and the target object in the imaging process.

[0158] Optionally, the first internal and external parameter information can include the vertical viewing angle of the camera, the image size of the image collected by the camera, and the optical center height of the camera.

[0159] The vertical viewing angle of the camera is the maximum viewing angle of the camera in the vertical direction, for example, 150 degrees.

[0160] The image size can be the image resolution, for example, 1024*768, which means that the image contains 1024 pixel points in the horizontal direction and 768 pixel points in the vertical direction. Optionally, the image resolution can include horizontal resolution and vertical resolution. Taking 1024*768 as an example, the horizontal resolution is 1024, and the vertical resolution is 768.

[0161] The optical center height of the camera is the distance between the optical center of the camera in the mobile robot and the motion plane of the mobile robot.

[0162] At this time, as shown in the following figure, the step S102B can include S102B1-S102B2. Figure 5

[0163] S102B1, based on the first distance, the vertical resolution of the frame image, and the vertical viewing angle of the camera, determining the down-view line included angle; wherein the down-view line included angle is the included angle between the down-view line and the optical axis of the camera, and the down-view line is the line connecting the optical center of the camera and the bottom of the target object.

[0164] As shown in the following figure, the step S102B1 can include S102B11-S102B12. Figure 6 ​As shown, it is a schematic diagram of global side view provided by the embodiment of the present application. The angle between the line connecting the optical center of the camera in the mobile robot and the bottom of the target object and the optical axis of the camera is the downward viewing angle.

[0165] In order to calculate the relative distance between the mobile robot and the target object, the downward viewing angle needs to be determined, so that the relative distance between the mobile robot and the target object can be calculated based on the downward viewing angle and the height of the optical center of the camera by using the trigonometric function.

[0166] In an implementation manner, the determination of the downward viewing angle based on the first distance, the vertical resolution of the frame image and the vertical viewing angle of the camera can include steps A1-A2:

[0167] In step A1, the ratio of the first distance to the vertical resolution is calculated as a first ratio.

[0168] In combination with Figure 4 and Figure 6 It can be known that the proportion of the first distance in the vertical direction of the image is positively correlated with the proportion of the downward viewing angle in the vertical viewing angle. When the proportion of the downward viewing angle in the vertical viewing angle is larger, the proportion of the first distance in the vertical direction of the image is also larger. Therefore, in order to determine the downward viewing angle, the ratio of the first distance to the vertical resolution can be calculated as a first ratio.

[0169] In step A2, the downward viewing angle is determined based on the first ratio and the vertical viewing angle.

[0170] Optionally, it can be approximately considered that the proportion of the first distance in the vertical direction of the image is linearly correlated with the proportion of the downward viewing angle in the vertical viewing angle. At this time, the product of the first ratio and the vertical viewing angle can be calculated as the downward viewing angle.

[0171] Alternatively, the present application provides another determination manner of the downward viewing angle. The target adjustment coefficient corresponding to the specified pixel point in the frame image can be determined according to the preset corresponding relationship between each pixel point and the adjustment coefficient, and the product of the adjustment coefficient and the first ratio is calculated as a second ratio, and the product of the second ratio and the vertical viewing angle is calculated as the downward viewing angle.

[0172] Wherein, each pixel point is a pixel point in the image collected by the camera, and the specified pixel point is a pixel point determined from the bottom pixel point of the target object.

[0173] The preset corresponding relationship between each pixel point and the adjustment coefficient can be determined by calibrating the camera in the mobile robot. For each pixel point, the corresponding adjustment coefficient can be the distance between the pixel point and the center pixel point, or the ratio of the distance between the actual object corresponding to the pixel point and the actual object corresponding to the center pixel point.

[0174] Therefore, when the downward line of sight intersection needs to be determined, an arbitrary pixel point among the pixel points on the bottom of the target object can be selected as a specified pixel point, and then the adjustment coefficient corresponding to the specified pixel point is taken as a target adjustment coefficient, the first ratio is adjusted by using the target adjustment coefficient, that is, the product of the adjustment coefficient and the first ratio is calculated as a second ratio, and then the product of the second ratio and the vertical angle of view is calculated as the downward line of sight angle.

[0175] In S102B2, the relative distance between the mobile robot and the target object when the frame image is collected is calculated based on the downward line of sight angle and the height of the optical center of the camera.

[0176] Optionally, it can be approximately considered that the triangle formed by the light ray of the camera on the mobile robot and the bottom of the target object is a right triangle, and at this time, the following formula can be used to calculate the relative distance between the mobile robot and the target object when the frame image is collected:

[0177] d=tan(90+m)*h

[0178] Wherein, d is the relative distance, and h is the height of the optical center of the camera.

[0179] Optionally, in order to accurately calculate the relative distance between the mobile robot and the target object when the frame image is collected, the vertical angle between the optical axis and the moving plane of the mobile robot can also be considered, and at this time, the following formula can be used to calculate the relative distance between the mobile robot and the target object when the frame image is collected:

[0180] d=tan(90+k-m)*h

[0181] Wherein, d is the relative distance, k is the vertical angle between the optical axis and the moving plane of the mobile robot, m is the downward line of sight angle, and h is the height of the optical center of the camera.

[0182] The vertical angle between the optical axis and the moving plane of the mobile robot can be determined by pre-calibration.

[0183] In the above scheme provided by the embodiment of the application, the contour information of the object in the environment can be determined. The downward line of sight angle can be calculated, and the relative distance between the mobile robot and the target object when the frame image is collected can be calculated based on the downward line of sight angle and the height of the optical center of the camera, thereby providing an implementation basis for determining the contour information of the object in the environment.

[0184] The object information determination method provided by another embodiment of the application can perform object type identification on the target object before step S101 is performed, and if the object type of the target object is a to-be-identified type, step S101 is performed. The to-be-identified type includes an unknown object type or a non-fixed shape object type.

[0185] There are several ways to perform object type recognition. For example, you can train a neural network model, select an object type recognition model, and then use the object type recognition model to process each frame of image after acquiring it, and determine the object category of the target object contained in the frame of image.

[0186] The aforementioned types to be identified include unknown object types or non-fixed-form object types. Unknown object types are those whose object types have not been identified. Non-fixed-form object types are those whose shapes are variable, such as socks and clothes, whose shapes are not fixed but variable.

[0187] If the object type of the target object is identified as the type to be identified, it means that the outline information of the target object needs to be determined, so that step S101 can be executed.

[0188] In this case, such as Figure 7 As shown, another embodiment of the object information determination method provided by the present invention may include steps S701-S707:

[0189] S701, perform object type recognition on the target object;

[0190] Specifically, regarding the target object, if it is a fixed-shape object such as a trash can or shoes, its outline can generally be determined through presets. Therefore, to reduce computational load, the outline information of such objects can be determined through presets. However, for unknown object types or non-fixed-shape object types, a more advanced method can be used. Figure 1 The method shown is used to determine the object information of the target object.

[0191] At this point, after identifying the target object, object type recognition can be performed. If the object type of the target object is the type to be recognized, then step S702 is executed; otherwise, if the object type of the target object is a fixed-shape object type, then step S705 is executed.

[0192] S702, Acquire multiple frames of images containing the target object during the process of the mobile robot performing a circling motion around the target object.

[0193] The implementation method of this step is the same as or similar to that of step S101. The implementation method can be found in the relevant description of step S101, and will not be repeated here.

[0194] S703, for each frame of the acquired image, identifies the relative distance between the mobile robot and the target object when the image of that frame was acquired;

[0195] The implementation of this step is the same as or similar to step S102, and the implementation can be referred to the related description of step S102, which is not described here.

[0196] S704, based on the relative distance between the mobile robot and the target object when collecting each frame of image and the position information in the world coordinate system, determining the contour information of the target object.

[0197] The implementation of this step is the same as or similar to step S103, and the implementation can be referred to the related description of step S103, which is not described here.

[0198] S705, determining the preset initial information corresponding to the object type of the target object as the to-be-utilized information; wherein, the initial information indicates the initial position of each edge point;

[0199] If the object type of the target object is a fixed form object type, the preset initial information corresponding to the object type of the target object can be determined. The initial information corresponding to each object type can include the initial position of each edge point of the object corresponding to the object type.

[0200] S706, acquiring the image collected by the mobile robot for the target object, and identifying the relative distance between the mobile robot and the target object when collecting the acquired image;

[0201] Optionally, only one frame of image collected by the mobile robot for the target object can be acquired in this step, and then based on the one frame of image, the relative distance between the mobile robot and the target object when collecting the acquired image is identified. The specific identification manner can be similar to step S102, and the specific implementation is referred to the related description of step S102.

[0202] S707, based on the identified relative position, adjusting the initial position of each edge point indicated by the to-be-utilized information to obtain the adjusted position of each edge point as the contour information of the target object.

[0203] Since the initial position of each edge point of the target object has been determined, after the relative distance between the mobile robot and the target object is determined, the initial position of each edge point indicated by the to-be-utilized information can be adjusted by using the identified relative position, so as to obtain the adjusted position of each edge point as the contour information of the target object.

[0204] The scheme provided by the embodiment of the present application can determine the contour information of the object in the environment. In the case that the object type of the target object is a fixed shape object type, the contour information of the target object can be determined based on the preset initial information, without the need of moving the robot to perform a surrounding movement for the target object and calculating the relative position for each frame of image, so that the process of determining the object information is simplified and the efficiency of determining the object information is improved.

[0205] The object information determination method provided by another embodiment of the present application can further determine the height of the target object based on the relative distance between the mobile robot and the target object when the frame of image is collected, the second distance and the second internal and external parameter information of the camera after identifying the relative distance between the mobile robot and the target object when the frame of image is collected.

[0206] The second distance is the pixel distance in the vertical direction between the top pixel point of the target object and the center pixel point of the frame of image.

[0207] As shown in Figure 4 , the second distance is the pixel distance in the vertical direction between the top pixel point of the target object and the center pixel point of the frame of image. The top pixel point can be any pixel point on the top line segment of the target object.

[0208] Any pixel point on the top line segment of the target object can be selected as the top pixel point, and then the top pixel coordinates of the top pixel point are determined, and then the center pixel coordinates of the center pixel point are determined. Then the difference between the vertical coordinates in the top pixel coordinates and the vertical coordinates in the center pixel coordinates can be calculated, and the absolute value of the difference is taken as the second distance. For example, the top pixel coordinates are (x3, y3) and the center pixel coordinates are (x4, y4), and the first distance is the absolute value of y3-y4.

[0209] Optionally, the second internal and external parameter information includes the vertical viewing angle of the camera and the vertical resolution of the image collected by the camera.

[0210] As shown in Figure 8 , the embodiment of the present application provides another schematic diagram of global side view. The angle between the optical axis and the upward line is n, so tan(n)=(h-x) / d, where x is the height of the target object, and x is the height of the target object. The angle n=θ*(dv2_pixels / V), dv2_pixels is the second distance, and V is the vertical resolution,

[0211] At this time, the height of the target object can be determined by the following formula:

[0212]

[0213] Wherein, x is the height of the target object, theta is the vertical visual angle, dv2_pixels is the second distance, V is the vertical resolution, and d is the relative distance between the mobile robot and the target object when the frame image is collected.

[0214] After the height of the target object is calculated, if the height of the target object is less than the preset height, the target object is determined as a crossable object, and if the height of the target object is not less than the preset height, the target object is determined as a non-crossable object and needs to be bypassed.

[0215] In the above scheme provided by the embodiment of the application, the contour information of the object in the environment can be determined. Moreover, the height of the target object can be calculated, so that the object information of the target object is more abundant.

[0216] The object information determination method provided by another embodiment of the application, after the plurality of frames of images containing the target object collected by the mobile robot in the process of the surrounding motion of the target object is acquired, the horizontal deflection angle of the mobile robot and the target object when the frame image is collected can be identified for each acquired frame of image.

[0217] Wherein, the horizontal deflection angle is the horizontal included angle between the horizontal line of sight and the optical axis of the camera in the mobile robot, and the horizontal line of sight is the line between the optical center of the camera and the outside of the target object. Optionally, the identification of the horizontal deflection angle of the mobile robot and the target object when the frame image is collected can include steps B1-B2, wherein:

[0218] Step B1, determining the pixel distance of the third pixel point and the fourth pixel point in the horizontal direction in the frame image as the third distance.

[0219] Wherein, the third pixel point is the pixel point on the outside of the target object, and the fourth pixel point is the center pixel point. At this time, an arbitrary pixel point can be selected from the outside line segment of the target object as the third pixel point, and then the third pixel coordinates of the third pixel point are determined, and the fourth pixel coordinates of the fourth pixel point are determined. Then the difference between the vertical coordinate in the third pixel coordinates and the horizontal coordinate in the fourth pixel coordinates can be calculated, and the absolute value of the difference is taken as the third distance. For example, the third pixel coordinates are (x5, y5), the fourth pixel coordinates are (x6, y6), and the third distance is the absolute value of x5-x6.

[0220] Step B2, based on the third distance, the horizontal resolution of the frame image, and the horizontal visual angle of the camera, the horizontal deflection angle of the mobile robot and the target object when the frame image is collected.

[0221] Similarly to the calculation of the downward angle, the ratio of the third distance to the horizontal resolution can be calculated as a third ratio, and then based on the third ratio and the horizontal angle, the horizontal deflection angle of the mobile robot relative to the target object when the frame image is collected can be determined.

[0222] In the above scheme provided by the embodiments of the present application, the contour information of the object in the environment can be determined. Moreover, the horizontal deflection angle can be determined, so that the object information of the target object is more abundant.

[0223] Corresponding to the object information determination method provided by the above embodiments of the present application, as shown in the figure, Figure 9 The embodiments of the present application also provide a mobile robot system, which comprises:

[0224] The image acquisition module 901 is configured to acquire a plurality of frames of images containing the target object collected by the mobile robot in the process of the surrounding motion of the target object.

[0225] The processor 902 is configured to, for each frame of acquired image, identify the relative distance between the mobile robot and the target object when the frame image is collected, and determine the contour information of the target object based on the relative distance between the mobile robot and the target object when each frame of image is collected and the position information in the world coordinate system.

[0226] It should be noted that the above image acquisition module 901 can be a camera, such as a monocular camera, a structured light depth camera, a binocular camera, a TOF depth camera, and a binocular stereo vision camera, etc.

[0227] Optionally, the processor 902, for each frame of acquired image, identifying the relative distance between the mobile robot and the target object when the frame image is collected, can comprise:

[0228] For each frame of acquired image, determining the pixel distance of the first pixel point and the second pixel point in the vertical direction in the frame image as a first distance, wherein the first pixel point is the bottom pixel point of the target object, and the second pixel point is the center pixel point.

[0229] Based on the first distance and the first internal and external parameter information of the camera in the mobile robot, the relative distance between the mobile robot and the target object when the frame image is collected is determined.

[0230] Optionally, the first internal and external parameter information comprises the vertical angle of the camera, the image size of the camera when collecting the frame image, and the optical center height of the camera.

[0231] The processor 902 determines the relative distance between the mobile robot and the target object when the frame image is captured based on the first distance and first internal and external parameter information of the camera in the mobile robot, which can include:

[0232] determining a down-view line angle based on the first distance, a vertical resolution of the frame image, and a vertical view angle of the camera, wherein the down-view line angle is an angle between a down-view line and an optical axis of the camera, and the down-view line is a line between an optical center of the camera and a bottom of the target object.

[0233] calculating the relative distance between the mobile robot and the target object when the frame image is captured based on the down-view line angle and an optical center height of the camera.

[0234] Optionally, the processor 902 determines the down-view line angle based on the first distance, the vertical resolution of the frame image, and the vertical view angle of the camera, which can include:

[0235] calculating a ratio of the first distance to the vertical resolution as a first ratio;

[0236] determining the down-view line angle based on the first ratio and the vertical view angle.

[0237] Optionally, the processor 902 determines the down-view line angle based on the first ratio and the vertical view angle, which can include:

[0238] calculating a product of the first ratio and the vertical view angle as the down-view line angle; or

[0239] determining a target adjustment coefficient corresponding to a specified pixel point in the frame image according to a preset corresponding relationship between each pixel point and an adjustment coefficient, calculating a product of the adjustment coefficient and the first ratio as a second ratio, and calculating a product of the second ratio and the vertical view angle as the down-view line angle, wherein the each pixel point is a pixel point in an image captured by the camera, and the specified pixel point is a pixel point determined from bottom pixel points of the target object.

[0240] Optionally, the processor 902 calculates the relative distance between the mobile robot and the target object when the frame image is captured based on the down-view line angle and the optical center height of the camera, which can include:

[0241] calculating the relative distance between the mobile robot and the target object when the frame image is captured using the following formula:

[0242] d = tan(90 + k - m) * h

[0243] Wherein, d is the relative distance, k is the vertical angle between the optical axis and the moving plane of the mobile robot, m is the downward line angle, and h is the optical center height of the camera.

[0244] Optionally, the processor 902 determines the contour information of the target object based on the relative distance between the mobile robot and the target object when each frame of image is captured and the position information in the world coordinate system, which can include:

[0245] For each frame of image, the position of a target edge point of the target object is determined based on the position information in the world coordinate system and the relative distance between the mobile robot and the target object when the frame of image is captured, as the edge position corresponding to the frame of image; wherein the target edge point is an edge point of the camera in the shooting range when the frame of image is captured.

[0246] The contour information of the target object is determined based on the edge positions corresponding to each frame of image.

[0247] Optionally, the processor 902 determines the contour information of the target object based on the edge positions corresponding to each frame of image, which can include:

[0248] The edge positions corresponding to each frame of image are taken as the contour information of the target object; or,

[0249] The edge positions corresponding to each frame of image are curve-fitted to obtain at least one fitting curve, and the positions of each point on the at least one fitting curve are determined, and the determined positions are taken as the contour information of the target object.

[0250] Optionally, before the processor 902 obtains the plurality of frames of images containing the target object collected in the process of the mobile robot performing the surrounding motion for the target object, the processor 902 can also be used for object type identification of the target object; if the object type of the target object is a to-be-identified type, the processor 902 executes the step of obtaining the plurality of frames of images containing the target object collected in the process of the mobile robot performing the surrounding motion for the target object; wherein the to-be-identified type includes an unknown object type or a non-fixed shape object type.

[0251] Optionally, the processor 902 is further configured to, if the object type of the target object is a fixed object type, determine preset initial information corresponding to the object type of the target object as the to-be-utilized information, wherein the initial information indicates initial positions of the edge points; acquire an image collected by the mobile robot for the target object, and identify a relative distance between the mobile robot and the target object when the image is collected; and based on the identified relative distance, adjust the initial positions of the edge points indicated by the to-be-utilized information to obtain adjusted positions of the edge points as the contour information of the target object.

[0252] Optionally, the manner in which the mobile robot performs the surrounding motion for the target object comprises:

[0253] In a case where the orientation of the camera of the mobile robot is irrelevant to the moving direction of the mobile robot, the mobile robot performs continuous circular motion for the target object; or,

[0254] In a case where the orientation of the camera of the mobile robot is relevant to the moving direction of the mobile robot, the mobile robot performs multi-segment arc motion for the target object.

[0255] Optionally, after identifying the relative distance between the mobile robot and the target object when the frame image is collected, the processor 902 is further configured to determine the height of the target object based on the relative distance between the mobile robot and the target object when the frame image is collected, a second distance, and second internal and external parameter information of the camera; wherein the second distance is a pixel distance in a vertical direction between a top pixel point of the target object in the frame image and a center pixel point of the frame image.

[0256] Optionally, the second internal and external parameter information comprises a vertical viewing angle of the camera and a vertical resolution of the image collected by the camera.

[0257] The processor 902 determines the height of the target object based on the relative distance between the mobile robot and the target object when the frame image is collected, the second distance, and the second internal and external parameter information of the camera can comprise:

[0258] The height of the target object is determined by using the following formula:

[0259]

[0260] wherein x is the height of the target object, θ is the vertical viewing angle, dv2_pixels is the second distance, V is the vertical resolution, and d is the relative distance between the mobile robot and the target object when the frame image is collected.

[0261] Optionally, the processor 902 is further configured to, after acquiring the plurality of images containing the target object collected by the mobile robot during the process of performing the surrounding motion on the target object, identify, for each acquired image, a horizontal deflection angle between the mobile robot and the target object when the mobile robot collects the image, wherein the horizontal deflection angle is a horizontal included angle between a horizontal line of sight and an optical axis of a camera in the mobile robot, and the horizontal line of sight is a line between an optical center of the camera and an outer side of the target object.

[0262] Optionally, in the embodiment shown in Figure 9 Optionally, in the embodiment shown in Figure 10 The embodiment of the present application further provides a mobile robot system, which comprises:

[0263] The power module 903 is configured to drive the mobile robot to perform the surrounding motion on the target object.

[0264] The power module 903 can be a moving component carried by the mobile robot, and can include a motor, a tire and the like.

[0265] Optionally, in the case where the orientation of the camera of the mobile robot is irrelevant to the moving direction of the mobile robot, the power module 903 can drive the mobile robot to perform a continuous circular motion on the target object; or, in the case where the orientation of the camera of the mobile robot is relevant to the moving direction of the mobile robot, the power module 903 can drive the mobile robot to perform a multi-segment arc motion on the target object.

[0266] In the above scheme provided by the embodiment of the present application, since the plurality of acquired images are collected by the mobile robot during the process of performing the surrounding motion on the target object, the plurality of acquired images are images of different orientations of the target object collected by the mobile robot, and the relative distance between the mobile robot and the target object when the mobile robot collects each image is the distance between the mobile robot and the contour edge of the target object when the mobile robot collects the image, and then the contour information of the target object can be determined in combination with the position information of the mobile robot in the world coordinate system when the mobile robot collects the image. It can be seen that, by the scheme, the contour information of the object in the environment can be determined.

[0267] Corresponding to the object information determination method provided by the above embodiment of the present application, as shown in Figure 11 The embodiment of the present application further provides an object information determination device, which comprises:

[0268] The image acquisition module 1101 is configured to acquire a plurality of images containing a target object, which are collected by a mobile robot during a surrounding motion for the target object.

[0269] The information calculation module 1102 is configured to identify, for each acquired image, a relative distance between the mobile robot and the target object when the image is collected.

[0270] The information determination module 1103 is configured to determine, based on the relative distance between the mobile robot and the target object when each image is collected and position information in a world coordinate system, contour information of the target object.

[0271] Optionally, the information calculation module comprises:

[0272] The first sub-module is configured to determine, for each acquired image, a pixel distance between a first pixel point and a second pixel point in a vertical direction of the image as a first distance, wherein the first pixel point is a bottom pixel point of the target object, and the second pixel point is a center pixel point.

[0273] The second sub-module is configured to determine, based on the first distance and first internal and external parameter information of a camera in the mobile robot, the relative distance between the mobile robot and the target object when the image is collected.

[0274] Optionally, the first internal and external parameter information comprises a vertical viewing angle of the camera, an image size of an image collected by the camera, and a principal point height of the camera.

[0275] The second sub-module comprises:

[0276] The included angle determination unit is configured to determine, based on the first distance, a vertical resolution of the image, and the vertical viewing angle of the camera, a downward viewing line included angle, wherein the downward viewing line included angle is an included angle between a downward viewing line and an optical axis of the camera, and the downward viewing line is a line connecting a principal point of the camera and a bottom of the target object.

[0277] The distance determination unit is configured to calculate, based on the downward viewing line included angle and the principal point height of the camera, the relative distance between the mobile robot and the target object when the image is collected.

[0278] Optionally, the included angle determination unit comprises:

[0279] The ratio calculation sub-unit is configured to calculate a ratio of the first distance to the vertical resolution as a first ratio.

[0280] The included angle determination sub-unit is configured to determine, based on the first ratio and the vertical viewing angle, the downward viewing line included angle.

[0281] Optionally, the included angle determination subunit is specifically configured to calculate a product of the first ratio and the vertical viewing angle as the downward line included angle; or, according to a preset corresponding relationship between each pixel point and an adjustment coefficient, determine a target adjustment coefficient corresponding to a specified pixel point in the frame image, calculate a product of the adjustment coefficient and the first ratio as a second ratio, and calculate a product of the second ratio and the vertical viewing angle as the downward line included angle; wherein the each pixel point is a pixel point in the image collected by the camera, and the specified pixel point is a pixel point determined from a bottom pixel point of the target object.

[0282] Optionally, the distance determination unit is specifically configured to calculate the relative distance between the mobile robot and the target object when collecting the frame image by using the following formula:

[0283] d=tan(90+k-m)*h

[0284] wherein d is the relative distance, k is a vertical included angle between the optical axis and a moving plane of the mobile robot, m is the downward line included angle, and h is the optical center height of the camera.

[0285] Optionally, the information determination module comprises:

[0286] A position determination sub-module is configured to, for each frame image, determine a position of a target edge point of the target object based on position information of the mobile robot in a world coordinate system when collecting the frame image and a relative distance between the mobile robot and the target object, as an edge position corresponding to the frame image; wherein the target edge point is an edge point of a shooting range of the camera when collecting the frame image.

[0287] An information determination sub-module is configured to determine contour information of the target object based on the edge positions corresponding to each frame image.

[0288] Optionally, the information determination sub-module is specifically configured to take the edge positions corresponding to each frame image as the contour information of the target object; or, perform curve fitting on the edge positions corresponding to each frame image to obtain at least one fitting curve, and determine positions of points on the at least one fitting curve, and take the determined positions as the contour information of the target object.

[0289] Optionally, the apparatus further comprises a type identification module configured to identify the object type of the target object before the image acquisition module acquires the plurality of images containing the target object collected by the mobile robot during the process of performing the surrounding motion on the target object; and if the object type of the target object is a type to be identified, the image acquisition module is invoked to acquire the plurality of images containing the target object collected by the mobile robot during the process of performing the surrounding motion on the target object; wherein the type to be identified includes an unknown object type or a non-fixed shape object type.

[0290] Optionally, the type identification module is further configured to, if the object type of the target object is a fixed shape object type, determine preset initial information corresponding to the object type of the target object as to-be-utilized information; acquire an image collected by the mobile robot on the target object, and identify a relative distance between the mobile robot and the target object when the acquired image is collected; and based on the identified relative position, adjust initial positions of edge points indicated by the to-be-utilized information to obtain adjusted positions of the edge points as contour information of the target object; wherein the initial information indicates the initial positions of the edge points.

[0291] Optionally, the manner in which the mobile robot performs the surrounding motion on the target object includes that, in a case where a camera of the mobile robot is irrelevant to a moving direction of the mobile robot, the mobile robot performs continuous circular motion on the target object; or in a case where the camera of the mobile robot is relevant to the moving direction of the mobile robot, the mobile robot performs multi-segment arc motion on the target object.

[0292] Optionally, the information calculation module is further configured to, after identifying the relative distance between the mobile robot and the target object when the frame image is collected, determine a height of the target object based on the relative distance between the mobile robot and the target object when the frame image is collected, a second distance, and second internal and external parameter information of the camera; wherein the second distance is a pixel distance in a vertical direction between a top pixel point of the target object in the frame image and a center pixel point of the frame image.

[0293] Optionally, the second internal and external parameter information includes a vertical viewing angle of the camera and a vertical resolution of the image collected by the camera.

[0294] The information calculation module comprises:

[0295] The height calculation sub-module is configured to determine the height of the target object by using the following formula:

[0296]

[0297] wherein x is the height of the target object, θ is the vertical viewing angle, dv2_pixels is the second distance, V is the vertical resolution, and d is the relative distance between the mobile robot and the target object when the frame image is captured.

[0298] Optionally, an angle identification module is configured to identify, for each frame of the acquired images, a horizontal deflection angle between the mobile robot and the target object when the frame image is captured after the image acquisition module acquires a plurality of images containing the target object in the process of the mobile robot performing the surrounding motion on the target object, wherein the horizontal deflection angle is a horizontal included angle between a horizontal line of sight and an optical axis of a camera in the mobile robot, and the horizontal line of sight is a line between an optical center of the camera and an outer side of the target object.

[0299] Optionally, the angle identification module comprises:

[0300] a distance determination sub-module configured to determine a pixel distance between a third pixel point and a fourth pixel point in the frame image in a horizontal direction as a third distance, wherein the third pixel point is an outer side pixel point with respect to the target object, and the fourth pixel point is a center pixel point.

[0301] an angle determination sub-module configured to determine, based on the third distance, a horizontal resolution of the frame image, and a horizontal viewing angle of the camera, the horizontal deflection angle between the mobile robot and the target object when the frame image is captured.

[0302] Optionally, the angle determination sub-module is specifically configured to calculate a ratio of the third distance to the horizontal resolution as a third ratio, and determine, based on the third ratio and the horizontal viewing angle, the horizontal deflection angle between the mobile robot and the target object when the frame image is captured.

[0303] In the above scheme provided by the embodiments of the present application, the acquired plurality of images are images captured by the mobile robot in the process of the mobile robot performing the surrounding motion on the target object, so that the acquired plurality of images are images of different orientations of the target object captured by the mobile robot, and the relative distance between the mobile robot and the target object when each frame of image is captured is the distance between the mobile robot and the contour edge of the target object when the frame image is captured, and then the contour information of the target object can be determined in combination with the position information of the mobile robot in the world coordinate system when the frame image is captured. It can be seen that the contour information of the object in the environment can be determined through the present scheme.

[0304] The embodiment of the present application further provides an electronic device, such as Figure 12 As shown in the figure, the electronic device comprises a processor 1201, a communication interface 1202, a memory 1203 and a communication bus 1204, wherein the processor 1201, the communication interface 1202 and the memory 1203 complete mutual communication through the communication bus 1204,

[0305] The memory 1203 is used for storing a computer program.

[0306] The processor 1201 is used for executing the program stored in the memory 1203, so as to realize the steps of the object information determination method provided in the above embodiment of the present application.

[0307] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0308] The communication interface is used for communication between the above electronic device and other devices.

[0309] The memory can comprise a Random Access Memory (RAM) and can also comprise a Non-Volatile Memory (NVM), for example at least one disk memory. Optionally, the memory can also be at least one storage device located away from the aforementioned processor.

[0310] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; can also be a Digital Signal Processing (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0311] In yet another embodiment provided by the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement the steps of any of the object information determination methods described above.

[0312] In yet another embodiment provided by the present application, a computer program product containing instructions which, when executed on a computer, cause the computer to perform any of the object information determination methods described in the above embodiments.

[0313] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium, or transferred from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) manner. The computer readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be magnetic media (for example, floppy disk, hard disk, magnetic tape), optical media (for example, DVD), or semiconductor media (for example, solid state disk (SSD)) and the like.

[0314] It should be noted that, in this document, the terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the processes, methods, articles or devices including a series of elements not only include those elements, but also include other elements not explicitly listed or inherent to such processes, methods, articles or devices. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0315] The various embodiments in the specification are described in a related manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the mobile robot system, the device, the electronic device, the computer readable storage medium and the computer program product embodiments are basically similar to the method embodiments, and the description is relatively simple. For related parts, refer to the part of the method embodiment.

[0316] The above only describes the preferred embodiments of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An object information determination method characterized by comprising: The method comprises: acquiring a plurality of images containing a target object collected by a mobile robot during a circling motion of the mobile robot relative to the target object; for each acquired image, identifying a relative distance between the mobile robot and the target object when the image is collected; based on the relative distance between the mobile robot and the target object when each image is collected and position information of the mobile robot in a world coordinate system, determining contour information of the target object; after the relative distance between the mobile robot and the target object when the image is collected is identified, the method further comprises: based on the relative distance between the mobile robot and the target object when the image is collected, a second distance, and second internal and external parameter information of a camera in the mobile robot, determining a height of the target object; wherein the second distance is a pixel distance in a vertical direction between a top pixel point of the target object in the image and a center pixel point of the image.

2. The method of claim 1, wherein, The method comprises: for each acquired image, determining a pixel distance in a vertical direction between a first pixel point and a second pixel point in the image as a first distance; wherein the first pixel point is a bottom pixel point of the target object, and the second pixel point is a center pixel point; based on the first distance and first internal and external parameter information of a camera in the mobile robot, determining the relative distance between the mobile robot and the target object when the image is collected.

3. The method of claim 2, wherein, The first internal and external parameter information comprises a vertical viewing angle of the camera, an image size of the image collected by the camera, and a principal point height of the camera. The method comprises: based on the first distance, a vertical resolution of the image, and the vertical viewing angle of the camera, determining a downward line-of-sight angle; wherein the downward line-of-sight angle is an angle between a downward line of sight and an optical axis of the camera, and the downward line of sight is a line connecting a principal point of the camera and a bottom of the target object; based on the downward line-of-sight angle and the principal point height of the camera, calculating the relative distance between the mobile robot and the target object when the image is collected.

4. The method of claim 3, wherein, The method comprises: calculating a ratio of the first distance and the vertical resolution as a first ratio; based on the first ratio and the vertical viewing angle, determining the downward line-of-sight angle.

5. The method of claim 4, wherein, The method comprises: calculating a product of the first ratio and the vertical viewing angle as the downward line-of-sight angle; or According to a preset corresponding relationship between each pixel point and an adjustment coefficient, a target adjustment coefficient corresponding to a specified pixel point in the frame image is determined, a product of the adjustment coefficient and the first ratio is calculated as a second ratio, and a product of the second ratio and the vertical viewing angle is calculated as a lower viewing angle.

6. The method of claim 3, wherein, The relative distance between the mobile robot and the target object when the frame image is captured is calculated based on the lower viewing angle and the height of the optical center of the camera, including: The relative distance between the mobile robot and the target object when the frame image is captured is calculated using the following formula: d = tan(90 + k - m) * h Wherein, d is the relative distance, k is the vertical angle between the optical axis and the moving plane of the mobile robot, m is the lower viewing angle, and h is the height of the optical center of the camera.

7. The method according to any one of claims 1 to 6, characterized in that, The profile information of the target object is determined based on the relative distance between the mobile robot and the target object when each frame image is captured and the position information in the world coordinate system, including: For each frame image, the position of a target edge point of the target object is determined based on the position information of the mobile robot in the world coordinate system when the frame image is captured and the relative distance to the target object, as the edge position corresponding to the frame image; wherein the target edge point is an edge point of the mobile robot located in the shooting range of the camera when the frame image is captured; The profile information of the target object is determined based on the edge positions corresponding to each frame image.

8. The method of claim 7, wherein, The profile information of the target object is determined based on the edge positions corresponding to each frame image, including: The edge positions corresponding to each frame image are taken as the profile information of the target object; or, Curve fitting is performed on the edge positions corresponding to each frame image to obtain at least one fitting curve, and the positions of each point on the at least one fitting curve are determined, and the determined positions are taken as the profile information of the target object.

9. The method according to any one of claims 1 to 6, characterized in that, Before obtaining the multiple frames of images containing the target object collected by the mobile robot during the process of circling the target object, the method further includes: Performing object type identification on the target object; If the object type of the target object is a to-be-identified type, performing the step of obtaining the multiple frames of images containing the target object collected by the mobile robot during the process of circling the target object; wherein the to-be-identified type includes an unknown object type or a non-fixed shape object type.

10. The method of claim 9, wherein, The method further includes: If the object type of the target object is a fixed shape object type, determining preset initial information corresponding to the object type of the target object as to-be-utilized information; wherein the initial information indicates the initial positions of each edge point. acquire an image collected by the mobile robot for the target object, and identify a relative distance between the mobile robot and the target object when the acquired image is collected; adjust initial positions of the edge points indicated by the to-be-utilized information based on the identified relative position, to obtain adjusted positions of the edge points as the contour information of the target object.

11. The method according to any one of claims 1 to 6, characterized in that, The manner in which the mobile robot performs the surrounding motion for the target object includes: in a case where a camera of the mobile robot is oriented regardless of a moving direction of the mobile robot, the mobile robot performs continuous circular motion for the target object; or in a case where the camera of the mobile robot is oriented in relation to the moving direction of the mobile robot, the mobile robot performs multi-segment arc motion for the target object.

12. The method of claim 1, wherein, The second internal and external parameter information includes a vertical angle of view of the camera and a vertical resolution at which the camera collects images; The determining of the height of the target object based on the relative distance between the mobile robot and the target object when the frame image is collected, the second distance, and the second internal and external parameter information of the camera includes: The height of the target object is determined by using the following formula: wherein x is the height of the target object, θ is the vertical angle of view, dv2_pixels is the second distance, V is the vertical resolution, and d is the relative distance between the mobile robot and the target object when the frame image is collected.

13. The method according to any one of claims 1 to 6, characterized in that, After the acquiring of the multiple frames of images collected by the mobile robot during the surrounding motion for the target object, the method further includes: For each acquired frame of image, identifying a horizontal deflection angle between the mobile robot and the target object when the frame of image is collected; wherein the horizontal deflection angle is a horizontal included angle between a horizontal line of sight and an optical axis of a camera in the mobile robot, and the horizontal line of sight is a line between an optical center of the camera and an outside of the target object.

14. A mobile robot system, characterized by The method includes: an image acquisition module, configured to acquire multiple frames of images collected by a mobile robot during surrounding motion for a target object, the multiple frames of images containing the target object; a processor, configured to, for each acquired frame of image, identify a relative distance between the mobile robot and the target object when the frame of image is collected; and determine contour information of the target object based on the relative distances between the mobile robot and the target object when the frames of images are collected and position information of the mobile robot in a world coordinate system. The processor is further configured to determine a height of the target object based on the relative distance between the mobile robot and the target object when the frame of image is collected, a second distance, and second internal and external parameter information of a camera in the mobile robot; wherein the second distance is a pixel distance in a vertical direction between a top pixel point of the target object in the frame of image and a center pixel point of the frame of image.

15. The mobile robotic system of claim 14, wherein, The method further includes: a power module, configured to drive the mobile robot to perform the surrounding motion for the target object.

16. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps of any one of claims 1-13. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps of any one of claims 1-13. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method steps of any one of claims 1-13.

17. A computer-readable storage medium, characterized in that, ​

Images