Method and apparatus for determining the orientation of an object

Abstract

The application provides a method and a device for determining the coordinate direction of an object, and belongs to the technical field of robot vision. The method for determining the coordinate direction of the object comprises the following steps: determining a target bounding rectangle of an object point cloud of a target object according to the object point cloud, wherein the target bounding rectangle comprises a first side and a second side that intersect; determining the ratio of the length of the first side to the length of the second side; and determining the coordinate direction of the target object according to the coordinate values of each point in the target bounding rectangle and the ratio.

Description

Technical Field

[0001] This application relates to the field of robot vision technology, and more specifically, to a method and apparatus for determining the coordinate orientation of an object. Background Technology

[0002] In related technologies, applications such as robot semantic navigation often require knowing the orientation of objects in the environment.

[0003] The orientation of an object is its posture in the real world. In related technologies, determining the orientation of an object requires methods such as template matching. Therefore, it is necessary to know the engineering drawing model of the object and determine the actual orientation of the object based on the orientation marked on the engineering drawing model.

[0004] In many scenarios, it is difficult to obtain an engineering drawing model of an object, making it difficult to determine the object's orientation. Summary of the Invention

[0005] This application aims to address at least one of the technical problems existing in the prior art or related technologies.

[0006] Therefore, the first aspect of this application proposes a method for determining the coordinate orientation of an object.

[0007] The second aspect of this application proposes a device for determining the coordinate orientation of an object.

[0008] A third aspect of this application proposes a readable storage medium.

[0009] The fourth aspect of this application proposes a computer program product.

[0010] The fifth aspect of this application proposes an electronic device.

[0011] The sixth aspect of this application proposes a robot.

[0012] In view of this, the first aspect of this application provides a method for determining the coordinate orientation of an object, comprising: determining a target bounding rectangle of the object point cloud based on the object point cloud of the target object, the target bounding rectangle including an intersecting first side and a second side; determining the ratio of the length of the first side to the length of the second side; and determining the coordinate orientation of the target object based on the coordinate values ​​and the ratio of each point within the target bounding rectangle.

[0013] The second aspect of this application provides an apparatus for determining the coordinate orientation of an object, comprising: a memory for storing programs or instructions; and a processor for executing programs or instructions to implement the steps of determining the coordinate orientation of an object as provided in any of the above technical solutions. Therefore, this apparatus for determining the coordinate orientation of an object also includes all the beneficial effects of the method for determining the coordinate orientation of an object as provided in any of the above technical solutions, and will not be repeated here to avoid repetition.

[0014] A third aspect of this application provides a readable storage medium having a program or instructions stored thereon. When the program or instructions are executed by a processor, they implement the steps of the method for determining the coordinate orientation of an object as provided in any of the above technical solutions. Therefore, the readable storage medium also includes all the beneficial effects of the method for determining the coordinate orientation of an object as provided in any of the above technical solutions. To avoid repetition, these effects will not be repeated here.

[0015] The fourth aspect of this application provides a computer program product stored in a storage medium. When executed by at least one processor, the computer program product implements the steps of the method for determining the coordinate orientation of an object as provided in any of the above technical solutions. Therefore, the computer program product also includes all the beneficial effects of the method for determining the coordinate orientation of an object as provided in any of the above technical solutions. To avoid repetition, these effects will not be repeated here.

[0016] The fifth aspect of this application provides an electronic device, including a means for determining the coordinate direction of an object as provided in any of the foregoing technical solutions; and / or a readable storage medium as provided in any of the foregoing technical solutions; and / or a computer program product as provided in any of the foregoing technical solutions, thus also including the means for determining the coordinate direction of an object as provided in any of the foregoing technical solutions; and / or a readable storage medium as provided in any of the foregoing technical solutions; and / or all the beneficial effects of the computer program product as provided in any of the foregoing technical solutions are not described in detail here to avoid repetition.

[0017] The sixth aspect of this application provides a robot, including a device for determining the coordinate direction of an object as provided in any of the above technical solutions; and / or a readable storage medium as provided in any of the above technical solutions; and / or a computer program product as provided in any of the above technical solutions, thus also including the device for determining the coordinate direction of an object as provided in any of the above technical solutions; and / or a readable storage medium as provided in any of the above technical solutions; and / or all the beneficial effects of the computer program product as provided in any of the above technical solutions are not described here to avoid repetition.

[0018] The embodiments of this application can determine the orientation of an object, that is, the coordinate direction of the object in the robot's visual coordinate system, based on the object point cloud. This process does not rely on the engineering drawing model of the object and can effectively identify the orientation of various objects, thereby improving the convenience of obtaining the object orientation and thus improving the robot's pathfinding performance. Attached Figure Description

[0019] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0020] Figure 1 A flowchart illustrating a method for determining the coordinate orientation of an object according to an embodiment of this application is shown;

[0021] Figure 2 A schematic diagram of a target enclosing rectangle according to an embodiment of this application is shown;

[0022] Figure 3 A schematic diagram of a sub-rectangle according to an embodiment of this application is shown;

[0023] Figure 4 A structural block diagram of an object coordinate orientation determination device according to an embodiment of this application is shown;

[0024] Figure 5 A structural block diagram of an object coordinate orientation determination device according to an embodiment of this application is shown. Detailed Implementation

[0025] To better understand the above-mentioned objectives, features, and advantages of this application, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0026] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Therefore, the scope of protection of this application is not limited to the specific embodiments disclosed below.

[0027] The following reference Figures 1 to 5 This application describes a method and apparatus for determining the coordinate orientation of an object according to some embodiments.

[0028] In some embodiments of this application, a method for determining the coordinate orientation of an object is provided. Figure 1 A flowchart illustrating a method for determining the coordinate orientation of an object according to an embodiment of this application is shown, such as... Figure 1 As shown, the method includes:

[0029] Step 102: Based on the object point cloud of the target object, determine the target bounding rectangle of the object point cloud. The target bounding rectangle includes the intersecting first side and second side.

[0030] Step 104: Determine the ratio of the length of the first side to the length of the second side;

[0031] Step 106: Determine the coordinate direction of the target object based on the coordinate values ​​of each point within the target's bounding rectangle and the ratio of the length of the first side to the length of the second side.

[0032] In the embodiments of this application, the robot includes a home robot, such as a robot vacuum cleaner or a service robot. The home robot can automatically perform certain tasks according to user instructions, such as delivering a TV remote control to a family member watching TV on the living room sofa. At this time, the user can place the TV remote control on the storage compartment of the home robot and give the instruction "deliver the TV remote control to the living room sofa". The robot will then automatically navigate to the "sofa" in the "living room" according to the robot's vision system.

[0033] During the robot's movement, it relies on the robot's vision system to identify routes and target objects. In the process, the robot uses image sensors and microwave radar sensors to build indoor point cloud data. By recognizing and processing the point cloud data, it plans its walking route and finds its destination.

[0034] Since furniture such as sofas and chairs have orientations, taking the scenario of the robot delivering a remote control as an example, in order to accurately deliver the remote control to the person sitting on the sofa, the robot needs to move to the front of the sofa, rather than the side or back. Therefore, the robot needs to identify the orientation of the sofa when delivering the remote control.

[0035] Specifically, embodiments of this application use robot vision sensors, such as image sensors and microwave radar sensors, to establish point cloud data of the robot's environment, including the point cloud data of the sofa, i.e., the target object.

[0036] After identifying the object point cloud of the sofa, the robot further determines the target bounding rectangle of the object point cloud. Specifically, the target bounding rectangle is the minimum bounding rectangle of the object point cloud. The target bounding rectangle includes a first side and a second side that intersects with the first side. If the lengths of the first side and the second side are not the same, then the first side and the second side correspond to the long side and the wide side of the minimum bounding rectangle, respectively.

[0037] After obtaining the target bounding rectangle of the target object, determine the ratio between the length of the first side of the target bounding rectangle and the length of the second side of the target bounding rectangle. This ratio is also the aspect ratio of the target bounding rectangle, and therefore can also express the aspect ratio of the target object.

[0038] According to prior knowledge, for some objects with significant characteristics in terms of aspect ratio, after defining the direction from the head to the tail of the object as the object's orientation, the object's orientation is often directly related to its aspect ratio. For example, the head and tail of a sofa are generally on the long side of the sofa, while the head and foot of a single bed are generally on the wide side of the single bed.

[0039] At the same time, these objects also have distinct shapes. For example, the rear of a sofa is defined as the seat cushion, and the head of a sofa is defined as the backrest. The height of the backrest is higher than the height of the seat cushion. Similarly, the head of a single bed is defined as the headboard, and the rear of a single bed is defined as the footboard. The height of the headboard is generally higher than the footboard.

[0040] This height information can be expressed using the z-axis coordinates of points in the object's point cloud.

[0041] Therefore, by obtaining the aspect ratio of the target object and the coordinate values ​​of the points in the object's point cloud, the orientation of the target object, that is, the coordinate direction of the target object in the robot's vision coordinate system, can be determined.

[0042] The embodiments of this application can determine the orientation of an object, that is, the coordinate direction of the object in the robot's visual coordinate system, based on the object point cloud. This process does not rely on the engineering drawing model of the object and can effectively identify the orientation of various objects, thereby improving the convenience of obtaining the object orientation and thus improving the robot's pathfinding performance.

[0043] Based on any of the above embodiments, the coordinate values ​​include x-axis coordinate values, y-axis coordinate values, and z-axis coordinate values; determining the target bounding rectangle of the object point cloud based on the object point cloud of the target object includes: obtaining the maximum value of the x-axis coordinate value and the maximum value of the y-axis coordinate value of each point in the object point cloud; determining the vertex coordinates of the target bounding rectangle based on the maximum value of the x-axis coordinate value and the maximum value of the y-axis coordinate value; and determining the target bounding rectangle based on the vertex coordinates.

[0044] In the embodiments of this application, the object point cloud determined based on robot vision includes multiple points, and the coordinate structure of these points is (x, y, z), that is, it includes x-axis coordinates, y-axis coordinates and z-axis coordinates.

[0045] After obtaining the object point cloud of the target object, the x-axis coordinates and y-axis coordinates of each point in the object point cloud are traversed sequentially, and the maximum and minimum values ​​of the x-axis and y-axis coordinates are determined.

[0046] Let the maximum value of the x-axis coordinate be x. max The minimum value of the x-axis coordinate is x min The maximum value of the y-axis coordinate is ymax The minimum value of the y-axis coordinate is y min Then we can obtain the target bounding rectangle of the object point cloud, that is, the vertex coordinates of the minimum bounding rectangle are: (x min y min ), (x min y max ), (x max y max ), (x max y min ).

[0047] Figure 2 A schematic diagram of a target enclosing rectangle according to an embodiment of this application is shown, such as... Figure 2 As shown, a rectangle 200 is defined by the above 4 vertices. This rectangle 200 is the minimum bounding rectangle of the object point cloud 202, which is also the target bounding rectangle.

[0048] This application determines the coordinate orientation of an object in the robot's visual coordinate system by using the minimum bounding rectangle of the object's point cloud, without relying on the object's engineering drawing model, thus improving the convenience of obtaining the object's orientation.

[0049] Based on any of the above embodiments, determining the ratio of the length of the first side to the length of the second side includes: determining the length of the first side and the length of the second side based on the vertex coordinates; and determining the ratio based on the length of the first side and the length of the second side.

[0050] In this embodiment, the vertex coordinates of the target bounding rectangle, i.e., the minimum bounding rectangle of the object point cloud, are respectively: (x min y min ), (x min y max ), (x max y max ), (x max y min ).

[0051] After obtaining the vertex coordinates, calculate (x max -x min ) Get the length of the first side, calculate (y max -y min The length of the second side is obtained. It is understood that the larger of the lengths of the first and second sides is the length of the longer side of the target rectangle, and the smaller of the lengths of the first and second sides is the length of the shorter side of the target rectangle.

[0052] After obtaining the lengths of the first and second sides, the ratio between the two is calculated to obtain the aspect ratio of the rectangle enclosing the target. By combining the orientation data of objects with different aspect ratios, the true orientation of the object is determined, thereby improving the robot's pathfinding performance.

[0053] Based on any of the above embodiments, the coordinate direction of the target object is determined according to the coordinate values ​​and ratios of each point within the target enclosing rectangle, including: dividing the target enclosing rectangle into multiple sub-rectangles, the sum of the areas of the multiple sub-rectangles being equal to the area of ​​the target enclosing rectangle; determining the average value of the z-axis coordinate values ​​of each point within every two adjacent sub-rectangles; determining the head coordinates and tail coordinates of the target object based on the average value and ratios; and determining the coordinate direction of the target object based on the head coordinates and tail coordinates.

[0054] In the embodiments of this application, based on prior knowledge, the orientation of some common indoor objects is strongly related to their shape. For example, the head of a sofa, which is the backrest, is significantly higher than the seat. Similarly, the head of a bed is significantly higher than the foot of the bed.

[0055] Therefore, in practical application scenarios, such as giving a TV remote control to someone sitting on a sofa, the orientation of the object can be determined based on the height of each part of the sofa, that is, the z-axis coordinate of each point.

[0056] Specifically, the minimum bounding rectangle of the target object, i.e., the target bounding rectangle, is divided into multiple sub-rectangles with equal or nearly equal areas. After obtaining multiple sub-rectangles, the z-axis coordinates of each point within the area of ​​each sub-rectangle are determined, and the average value of these z-axis coordinates is calculated. This average value can intuitively reflect the height of the target object within a sub-rectangle area.

[0057] The height of the target object on a certain side can be determined by averaging the z-axis coordinates within two adjacent sub-rectangles. Therefore, after obtaining the average z-axis coordinates of each point in every two adjacent sub-matrices, the object orientation, i.e., the coordinate direction of the target object in the robot's vision coordinate system, can be determined by the aspect ratio of the target object (i.e., the ratio of the length of the first side to the length of the second side) and the height of the target object on each side (i.e., the first and second sides).

[0058] Taking a sofa as an example, when it is determined that one long side of the target object is significantly higher than the other long side, the longer side with the higher height is determined as the head of the sofa, and the longer side with the lower height is determined as the tail of the sofa. Thus, the orientation of the sofa, that is, the coordinate direction of the sofa, is determined according to the direction of the head and tail vectors.

[0059] This application determines the orientation, or coordinate direction, of the target object by dividing the minimum bounding rectangle of the target object into multiple adjacent sub-rectangles and combining the aspect ratio of the minimum bounding rectangle. This process does not rely on the engineering drawing model of the object and can determine the object's orientation more quickly.

[0060] Based on any of the above embodiments, the target enclosing rectangle is divided into multiple adjacent sub-rectangles, including: determining a sub-rectangle with the intersection of the first side and the second side, the midpoint of the first side, the midpoint of the second side, and the center point of the target enclosing rectangle as vertices.

[0061] In the embodiments of this application, the target bounding rectangle, that is, the minimum bounding rectangle of the target object, includes an intersecting first side and a second side, wherein the intersection point of the first side and the second side is a vertex of the minimum bounding rectangle.

[0062] Taking the first side as the long side of the target enclosing rectangle and the second side as the wide side of the target enclosing rectangle as an example, the first line segment is determined by taking the midpoint of the two long sides of the target enclosing rectangle as the endpoint, and the second line segment is determined by taking the midpoint of the two wide sides of the target enclosing rectangle as the endpoint. The intersection of the first line segment and the second line segment is the midpoint of the target enclosing rectangle.

[0063] Figure 3 A schematic diagram of a sub-rectangle according to an embodiment of this application is shown, such as... Figure 3 As shown, a sub-rectangle is defined using a vertex of the target-enclosing rectangle, the midpoint of the adjacent long and short sides, and the midpoint of the target-enclosing rectangle as vertices. This results in four sub-rectangles with equal or nearly equal areas, effectively dividing the target-enclosing rectangle into four equal sub-rectangles. Each of these four sub-rectangles has an adjacent sub-rectangle on the short side of the target-enclosing rectangle, and each sub-rectangle also has an adjacent sub-rectangle on the long side of the target-enclosing rectangle.

[0064] Therefore, by using the average z-axis coordinates of each point in every two adjacent sub-rectangles, the height of the target object on the width and length sides can be determined, and thus the orientation of the target object can be identified.

[0065] This application divides the target's enclosing rectangle into four sub-rectangles of equal area, which are adjacent on both the width and length sides. The coordinate direction of the target object is determined by the average value of the z-axis coordinates of these sub-rectangles, thus achieving a quick and convenient method for determining the object's orientation.

[0066] Based on any of the above embodiments, determining the head coordinates and tail coordinates of the target object according to the average value and the ratio includes: when the ratio is less than a preset threshold, determining the adjacent first sub-rectangle and second sub-rectangle corresponding to the maximum value of the average value; determining the head coordinates according to the coordinates of the first intersection point of the common side of the first sub-rectangle and the second sub-rectangle and the side of the target enclosing rectangle; and determining the tail coordinates according to the coordinates of the midpoint of the side opposite the side where the first intersection point is located in the side of the target enclosing rectangle.

[0067] In this embodiment, a preset threshold, which is used to determine the shape of the target object, is a constant less than 1 and greater than 0 in some implementations. If the ratio of the length of the first side of the target object to the length of the second side of the target object, that is, the aspect ratio of the target object, is less than the preset threshold, it indicates that the shape of the target object is relatively square, and the long side and the wide side of the minimum enclosing rectangle of the target object are relatively close. At this time, based on prior knowledge, all four sides of the minimum enclosing rectangle of the target object may be the "head" or "tail" of the target object.

[0068] Meanwhile, the target enclosing rectangle is divided into 4 sub-rectangles. These 4 sub-rectangles consist of 2 adjacent sub-rectangles on the long side of the target enclosing rectangle and 2 adjacent sub-rectangles on the wide side of the target enclosing rectangle, for a total of 4 sets of adjacent sub-rectangles.

[0069] Among these four groups of adjacent sub-rectangles, the group with the highest average z-axis coordinate value is determined. This group of adjacent sub-rectangles specifically includes the first sub-rectangle and the second sub-rectangle. The first intersection point of the common side of the first and second sub-rectangles with its corresponding long or wide side is defined as the "head point" of the target object, and its coordinates are recorded as the head coordinates of the target object. The midpoint of the side opposite the first intersection point among the four sides of the target's enclosing rectangle is defined as the "tail point" of the target object, and its coordinates are recorded as the tail coordinates of the target object.

[0070] After obtaining the head and tail coordinates, let the head coordinates be (x1, y1) and the tail coordinates be (x2, y2). Then, the coordinate direction of the target object can be calculated using the following formula:

[0071]

[0072] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0073] This application embodiment determines the head and tail coordinates of the target object, determines the object's coordinate direction based on the head and tail coordinates, and then determines the object's orientation, thereby improving the pathfinding performance based on robot vision.

[0074] Based on any of the above embodiments, when the ratio is greater than or equal to a preset threshold, the third side of the target enclosing rectangle is determined according to the object type of the target object; the adjacent third and fourth sub-rectangles corresponding to the maximum value of the average value are determined, wherein the third and fourth sub-rectangles are adjacent on the third side; the head coordinates are determined according to the coordinates of the second intersection point of the common side of the third and fourth sub-rectangles and the third side; and the tail coordinates are determined according to the coordinates of the midpoint of the side opposite the side where the second intersection point is located in the target enclosing rectangle.

[0075] In this application embodiment, a preset threshold, which is a threshold used to determine the shape of the target object, is, in some embodiments, a constant less than 1 and greater than 0. If the ratio of the length of the first side of the target object to the length of the second side of the target object, that is, the aspect ratio of the target object, is greater than or equal to the preset threshold, it indicates that the difference between the length of the long side and the length of the wide side of the minimum enclosing rectangle of the target object is large.

[0076] Based on prior knowledge, for scenarios where the target object is a sofa, the two longest sides of the minimum bounding rectangle of the target object are more likely to be the "head" or "tail" of the target object. For scenarios where the target object is a single bed, the two shortest sides of the minimum bounding rectangle of the target object are more likely to be the "head" or "tail" of the target object.

[0077] At this point, based on the object type of the target object, it is determined that the head and tail of this type of object are generally on the long side or the wide side. Taking the target object as a sofa as an example, the head of the sofa generally refers to the back of the sofa, and the back of the sofa is generally on the long side of the minimum bounding rectangle of the object point cloud of the sofa. Therefore, the long side of the target bounding rectangle can be recorded as the third side, and in the four sub-rectangles divided by the minimum bounding rectangle, two sets of two adjacent sub-rectangles on the long side, that is, the third side, are determined.

[0078] It is understandable that if the target object is a single bed, and the headboard and footboard of the single bed are on the wide side of the minimum bounding rectangle of the object's point cloud, then the wide side of the target's bounding rectangle is denoted as the third side.

[0079] Among the two adjacent sub-rectangles on the third side, determine the pair of adjacent sub-rectangles with the highest average z-axis coordinate values. This pair of adjacent sub-rectangles specifically includes the third and fourth sub-rectangles. The third and fourth sub-rectangles are adjacent on the third side. The second intersection point of the common side of the third and fourth sub-rectangles with the third side is determined as the "head point" of the target object, and the coordinates of this second intersection point are taken as the head coordinates of the target object. The midpoint of the side opposite the second intersection point among the four sides of the target's enclosing rectangle is recorded as the "tail point" of the target object, and its coordinates are taken as the tail coordinates of the target object.

[0080] After obtaining the head and tail coordinates, let the head coordinates be (x1, y1) and the tail coordinates be (x2, y2). Then, the coordinate direction of the target object can be calculated using the following formula:

[0081]

[0082] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0083] This application embodiment determines the head and tail coordinates of the target object, determines the object's coordinate direction based on the head and tail coordinates, and then determines the object's orientation, thereby improving the pathfinding performance based on robot vision.

[0084] Based on any of the above embodiments, before determining the third side of the target enclosing rectangle according to the object type of the target object, the method further includes:

[0085] Acquire image data of the target object;

[0086] Determine the object type based on the image data;

[0087] The third side of the target's bounding rectangle is determined based on the target object's object type, including:

[0088] Based on the object type, determine the third side from a pre-set database, where the third side is either the longer or shorter side of the rectangle enclosing the target; or

[0089] Display image data and the target bounding rectangle;

[0090] Receive selection input for the third side of the rectangle enclosing the target;

[0091] In response to the selected input, determine the third side.

[0092] In this embodiment of the application, the robot vision system further includes an image sensor, which captures an image of the target object and uses image recognition technology to determine the object type.

[0093] In some embodiments, when determining the third side, based on the object type, a pre-defined database is consulted to determine whether the head or tail of the corresponding type of object lies on the long side or the short side. For example, for objects such as sofas, the head or tail is generally on the long side, while for objects such as single beds, the head or tail is generally on the short side. Therefore, a database corresponding to different object types can be established in advance, and the edges containing the head and tail coordinates can be determined based on the database, thereby determining the coordinate direction of the target object.

[0094] In other embodiments, the user can manually specify the long or short side of the target object as the third side, that is, the user determines the head and tail of the target object, thereby determining the coordinate direction of the target object. When the robot carries the object, the user can specify the head and tail of the target object, which can enable the robot to accurately walk to the target position, such as sending the remote control to the side of the sofa cushion, thus improving the robot's pathfinding performance.

[0095] Based on any of the above embodiments, the first side includes a first long side and a second long side, and the second side includes a first wide side and a second wide side;

[0096] Determining the head and tail coordinates of the target object based on averages and ratios includes: determining a first difference between a first average and a second average, wherein the first average is the average of the z-coordinates of points within two adjacent sub-rectangles on the first long side, and the second average is the average of the z-coordinates of points within two adjacent sub-rectangles on the second long side; determining a second difference between a third average and a fourth average, wherein the third average is the average of the z-coordinates of points within two adjacent sub-rectangles on the first wide side, and the fourth average is the average of the z-coordinates of points within two adjacent sub-rectangles on the second wide side; and determining a second difference between the first and second averages. If the first difference is greater than the second difference, the coordinates of the third intersection point of the common edge of the two sub-rectangles corresponding to the larger of the first and second average values ​​and the edge of the target enclosing rectangle are determined as the head coordinates; the tail coordinates are determined based on the coordinates of the midpoint of the side opposite the edge containing the third intersection point of the target enclosing rectangle. If the first difference is less than the second difference, the coordinates of the fourth intersection point of the common edge of the two sub-rectangles corresponding to the larger of the third and fourth average values ​​and the edge of the target enclosing rectangle are determined as the head coordinates; the tail coordinates are determined based on the coordinates of the midpoint of the side opposite the edge containing the fourth intersection point of the target enclosing rectangle.

[0097] In this embodiment, the four sides of the target enclosing rectangle are specifically a first long side, a first wide side, a second long side, and a second wide side. The target enclosing rectangle is divided into four sub-rectangles of equal area. Among the four sub-rectangles, there are two sets of two adjacent sub-rectangles on the long side of the target enclosing rectangle, and two sets of two adjacent sub-rectangles on the wide side of the target enclosing rectangle.

[0098] Specifically, the average z-axis coordinates of each point within two adjacent sub-rectangles on the long side of the target's enclosing rectangle are denoted as the first average and the second average. The average z-axis coordinates of each point within two adjacent sub-rectangles on the wide side of the target's enclosing rectangle are denoted as the third average and the fourth average.

[0099] Calculate the difference between the first average and the second average, and denote it as the first difference. Calculate the difference between the third average and the fourth average, and denote it as the second difference.

[0100] If the first difference is greater than the second difference, it indicates that the height difference between the two longer sides of the target object is greater. If the first difference is less than the second difference, it indicates that the height difference between the two shorter sides of the target object is greater.

[0101] Based on prior knowledge, for common indoor objects such as sofas and beds, the height difference between the head and tail is generally significantly greater than the height difference between the two sides. For example, the two armrests of a sofa are the same height, and the two sides of a bed are also the same height.

[0102] Therefore, if the first difference is greater than the second difference, then among the two groups of two sub-rectangles that are adjacent on the long side, the group of two sub-rectangles with the larger average value of the z-axis coordinates is determined. The third intersection point of the common side of these two sub-rectangles with the long side of the target enclosing rectangle is determined as the head point, and the coordinates of the third intersection point are taken as the head coordinates. The midpoint of the side opposite the side where the third intersection point is located among the four sides of the target enclosing rectangle is determined as the tail point, and its coordinates are taken as the tail coordinates.

[0103] If the first difference is less than the second difference, then among the two groups of two adjacent sub-rectangles on the wide side, determine the group of two sub-rectangles with the larger average z-axis coordinate value. The fourth intersection point of the common side of these two sub-rectangles with the wide side of the target enclosing rectangle is determined as the head point, and the coordinates of the fourth intersection point are taken as the head coordinates. The midpoint of the side opposite the fourth intersection point among the four sides of the target enclosing rectangle is determined as the tail point, and its coordinates are taken as the tail coordinates.

[0104] After obtaining the head and tail coordinates, let the head coordinates be (x1, y1) and the tail coordinates be (x2, y2). Then, the coordinate direction of the target object can be calculated using the following formula:

[0105]

[0106] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0107] This application embodiment determines the head and tail coordinates of the target object, determines the object's coordinate direction based on the head and tail coordinates, and then determines the object's orientation, thereby improving the pathfinding performance based on robot vision.

[0108] Based on any of the above embodiments, determining the average value of the z-axis coordinates of each point within every two adjacent sub-rectangles includes: determining the N largest z-coordinate values ​​in each of the two adjacent sub-rectangles, where N is a positive integer; and determining the average value based on the N z-coordinate values.

[0109] In this embodiment of the application, when determining the average value of the z-axis coordinates of each point within two adjacent sub-rectangles, the N largest z-coordinate values ​​can be determined from all points in the region of these two adjacent sub-rectangles. Specifically, all points in this region can be sorted according to the order of their z-axis coordinates, and the first N points after sorting can be retained. The average value can be determined based on the z-axis coordinates of these N points.

[0110] It is understandable that if the number of points in the region is less than N, the average value is determined based on the z-axis coordinates of all points in the region.

[0111] This application calculates the average value of the N largest z-axis coordinates within a region to describe the height of the target object within that region. This more accurately reflects the actual height of the object and improves the accuracy of determining the object's orientation, i.e., the coordinate direction of the target object.

[0112] Based on any of the above embodiments, determining the coordinate direction of the target object according to the head coordinates and tail coordinates includes:

[0113] Determine the orientation angle of the target object based on its head and tail coordinates;

[0114] Determine the coordinate direction based on the orientation angle;

[0115] The orientation angle is calculated using the following formula:

[0116]

[0117] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0118] In this embodiment of the application, after obtaining the head coordinates and tail coordinates, let the head coordinates be (x1, y1) and the tail coordinates be (x2, y2). Then, the coordinate direction of the target object can be calculated using the following formula:

[0119]

[0120] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0121] The orientation angle is the angle between the direction vector of the target object and the standard direction vector. Once the orientation angle is obtained, the coordinate direction of the target object can be determined.

[0122] This application embodiment determines the head and tail coordinates of the target object, determines the object's coordinate direction based on the head and tail coordinates, and then determines the object's orientation, thereby improving the pathfinding performance based on robot vision.

[0123] Based on any of the above embodiments, the maximum value of the z-axis coordinate of the intersection point of the first straight line and the object point cloud is greater than the maximum value of the z-axis coordinate of the intersection point of the second straight line and the object point cloud.

[0124] The first line passes through the head coordinate and is perpendicular to both the x-axis and y-axis, while the second line passes through the tail coordinate and is perpendicular to both the x-axis and y-axis.

[0125] In the embodiments of this application, furniture objects such as sofas generally have shape characteristics. For example, the rear of the sofa is defined as the seat cushion, and the head of the sofa is defined as the backrest. The height of the backrest is higher than the height of the seat cushion. Similarly, the head of a single bed is defined as the headboard, and the rear of the single bed is defined as the footboard. The height of the headboard is generally higher than the footboard.

[0126] Therefore, by using the aforementioned prior knowledge as a feature to identify the coordinate direction of objects, the direction of common household furniture such as sofas can be accurately identified. By using the direction of the furniture to control the robot's movement, the robot's pathfinding performance can be improved.

[0127] In some embodiments of this application, the length, width, and height of the object's point cloud are aligned with the coordinate system, and the height direction is aligned with the Z-axis.

[0128] The orientation of an object is defined by the angle between the vector AB formed by the object's head point A and tail point B, and the X-axis, ranging from -180° to 180°. For some objects, the head is generally higher than the tail, such as beds and sofas. For objects with a large length-to-width ratio, such as beds, the headboard is usually on the shorter side, while the headboard of a sofa is usually on the longer side. Therefore, combining the length-to-width ratio can improve the accuracy of determining the head and tail of an object.

[0129] The specific method is as follows:

[0130] Find the minimum bounding rectangle of the object's point cloud to obtain the object's aspect ratio:

[0131] Iterate through the x and y coordinates of each point in the object's point cloud, and find the maximum and minimum values ​​of the x and y coordinates respectively, let's say x. max x min y max y min The coordinates of the four vertices of the minimum bounding rectangle of the object's point cloud are: (x... min y min ), (x min y max ), (x max y max ), (x max y min ).

[0132] Take x max -x min and y max -y min If the larger value in the equation is the length L of the object and the smaller value is the width W of the object, then the aspect ratio S of the object is S = L ÷ W.

[0133] Divide the rectangle into four equal regions:

[0134] Assume the midpoints of the four sides of a rectangle are point 1, point 2, point 3, and point 4. Based on the coordinates of the four vertices of the rectangle, calculate the coordinates of the midpoints of the four sides. Connect the midpoints of opposite sides to divide the rectangle into four equal regions A, B, C, and D. Figure 3 As shown. Each region is a rectangle, and based on the previous information, it is easy to obtain the vertex coordinates of the rectangle corresponding to each region.

[0135] Each point in the point cloud is divided into four regions:

[0136] For each region, find the maximum and minimum values ​​of the coordinates of the four vertices of its corresponding rectangle along the x-axis and y-axis, respectively. Let's assume they are x... max x min y max y min If the coordinates of a point in a point cloud are (x, y, z), and its coordinates satisfy x min <x<x max And y min <y<y max If the point is in the specified region, then that point should be assigned to that region. For each point in the point cloud, assign it to the corresponding region using the method described above.

[0137] Sort the points in each region by their z-coordinate from largest to smallest, and keep the first N points:

[0138] When the number of points in each region is greater than N, sort the points in each region by their z-coordinate from largest to smallest and keep the first N points. When the number of points in each region is less than N, do not perform any processing.

[0139] Find the average z-coordinate of all points in the adjacent region:

[0140] like Figure 3 As shown, there are a total of 4 adjacent regions: region A and region B, region B and region C, region C and region D, and region D and region A. For ease of description later, these adjacent regions are abbreviated as: adjacent region AB, adjacent region BC, adjacent region CD, and adjacent region DA. Find the average z-coordinate of all points in the adjacent regions.

[0141] Calculate the coordinates of the corresponding points at the head and tail of the object by combining the aspect ratio and the average z-coordinate.

[0142] Conclusion 1: The head and tail of an object are among the midpoints of the four sides of the minimum bounding rectangle of the object's point cloud. If it is known that the head of the object is at the midpoint of one of the sides, then the tail of the object is the midpoint of the opposite side. For example, if the head of the object is point 3, then the tail of the object is point 1.

[0143] Each adjacent region is associated with the midpoint of one of the sides of the minimum bounding rectangle of the object's point cloud. The associated points of adjacent regions AB, BC, CD, and DA are: point 2, point 3, point 4, and point 1 respectively.

[0144] Method 1:

[0145] When the aspect ratio of the object is less than k (1 < k), the possible number of positions of the object's head is 4. Obtain the adjacent region corresponding to the largest average value of the Z coordinates. The associated point of this adjacent region is the object's head, and the object's tail can be obtained according to Conclusion 1.

[0146] When the aspect ratio of the object is greater than k, combine the aspect ratio of the object and the possible positions of the head to determine the possible positions of the object's head. At this time, the possible number of positions of the object's head is 2.

[0147] If the head of the object is generally located on the long side of the minimum bounding rectangle of the object's point cloud, such as a sofa, then select the adjacent region whose associated point is on the long side of the minimum bounding rectangle of the object's point cloud from all adjacent regions. Obtain the adjacent region corresponding to the largest average value of the z coordinates among them. The associated point of this adjacent region is the object's head, and the object's tail can be obtained according to Conclusion 1.

[0148] If the head of the object is generally located on the short side of the minimum bounding rectangle of the object's point cloud, such as a bed, then select the adjacent region whose associated point is on the short side of the minimum bounding rectangle of the object's point cloud from all adjacent regions. Obtain the adjacent region corresponding to the largest average value of the Z coordinates among them. The associated point of this adjacent region is the object's head, and the object's tail can be obtained according to Conclusion 1.

[0149] Method 2:

[0150] Objects in ordinary families are relatively symmetrical. If the object is symmetrical, then the average values of the Z coordinates of the two adjacent regions obtained by being divided by the line connecting the head and tail points should not differ much. Combining this prior knowledge, calculate the absolute value of the difference between the average values of the Z coordinates of adjacent regions AB and CD and the absolute value of the difference between the average values of the Z coordinates of adjacent regions BC and DA, and determine which absolute value is larger. For the two adjacent regions corresponding to the larger absolute value, determine which adjacent region has the largest average value of the Z coordinates. The associated point of this adjacent region is the object's head, and the object's tail can be obtained according to Conclusion 1.

[0151] After obtaining the head and tail coordinates, let the head coordinates be (x1, y1) and the tail coordinates be (x2, y2). Then, the coordinate direction of the target object can be calculated using the following formula:

[0152]

[0153] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0154] In some embodiments of this application, a device for determining the coordinate orientation of an object is provided. Figure 4 A structural block diagram of an object coordinate orientation determination device according to an embodiment of this application is shown, such as... Figure 4 As shown, the object's coordinate orientation determination device 400 includes:

[0155] The determination module 402 is used to: determine the target bounding rectangle of the object point cloud based on the object point cloud of the target object, the target bounding rectangle including the intersecting first side and second side; determine the ratio of the length of the first side to the length of the second side; and determine the coordinate direction of the target object based on the coordinate values ​​and ratio of each point within the target bounding rectangle.

[0156] In the embodiments of this application, the robot includes a home robot, such as a robot vacuum cleaner or a service robot. The home robot can automatically perform certain tasks according to user instructions, such as delivering a TV remote control to a family member watching TV on the living room sofa. At this time, the user can place the TV remote control on the storage compartment of the home robot and give the instruction "deliver the TV remote control to the living room sofa". The robot will then automatically navigate to the "sofa" in the "living room" according to the robot's vision system.

[0157] During the robot's movement, it relies on the robot's vision system to identify routes and target objects. In the process, the robot uses image sensors and microwave radar sensors to build indoor point cloud data. By recognizing and processing the point cloud data, it plans its walking route and finds its destination.

[0158] Since furniture such as sofas and chairs have orientations, taking the scenario of the robot delivering a remote control as an example, in order to accurately deliver the remote control to the person sitting on the sofa, the robot needs to move to the front of the sofa, rather than the side or back. Therefore, the robot needs to identify the orientation of the sofa when delivering the remote control.

[0159] Specifically, embodiments of this application use robot vision sensors, such as image sensors and microwave radar sensors, to establish point cloud data of the robot's environment, including the point cloud data of the sofa, i.e., the target object.

[0160] After identifying the object point cloud of the sofa, the robot further determines the target bounding rectangle of the object point cloud. Specifically, the target bounding rectangle is the minimum bounding rectangle of the object point cloud. The target bounding rectangle includes a first side and a second side that intersects with the first side. If the lengths of the first side and the second side are not the same, then the first side and the second side correspond to the long side and the wide side of the minimum bounding rectangle, respectively.

[0161] After obtaining the target bounding rectangle of the target object, determine the ratio between the length of the first side of the target bounding rectangle and the length of the second side of the target bounding rectangle. This ratio is also the aspect ratio of the target bounding rectangle, and therefore can also express the aspect ratio of the target object.

[0162] According to prior knowledge, for some objects with significant characteristics in terms of aspect ratio, after defining the direction from the head to the tail of the object as the object's orientation, the object's orientation is often directly related to its aspect ratio. For example, the head and tail of a sofa are generally on the long side of the sofa, while the head and foot of a single bed are generally on the wide side of the single bed.

[0163] At the same time, these objects also have distinct shapes. For example, the rear of a sofa is defined as the seat cushion, and the head of a sofa is defined as the backrest. The height of the backrest is higher than the height of the seat cushion. Similarly, the head of a single bed is defined as the headboard, and the rear of a single bed is defined as the footboard. The height of the headboard is generally higher than the footboard.

[0164] This height information can be expressed using the z-axis coordinates of points in the object's point cloud.

[0165] Therefore, by obtaining the aspect ratio of the target object and the coordinate values ​​of the points in the object's point cloud, the orientation of the target object, that is, the coordinate direction of the target object in the robot's vision coordinate system, can be determined.

[0166] The embodiments of this application can determine the orientation of an object, that is, the coordinate direction of the object in the robot's visual coordinate system, based on the object point cloud. This process does not rely on the engineering drawing model of the object and can effectively identify the orientation of various objects, thereby improving the convenience of obtaining the object orientation and thus improving the robot's pathfinding performance.

[0167] Based on any of the above embodiments, the coordinate values ​​include x-axis coordinate values, y-axis coordinate values, and z-axis coordinate values; the determination module is specifically used for: obtaining the maximum value of the x-axis coordinate value and the maximum value of the y-axis coordinate value of each point in the object point cloud; determining the vertex coordinates of the target enclosing rectangle based on the maximum value of the x-axis coordinate value and the maximum value of the y-axis coordinate value; and determining the target enclosing rectangle based on the vertex coordinates.

[0168] In the embodiments of this application, the object point cloud determined based on robot vision includes multiple points, and the coordinate structure of these points is (x, y, z), that is, it includes x-axis coordinates, y-axis coordinates and z-axis coordinates.

[0169] After obtaining the object point cloud of the target object, the x-axis coordinates and y-axis coordinates of each point in the object point cloud are traversed sequentially, and the maximum and minimum values ​​of the x-axis and y-axis coordinates are determined.

[0170] Let the maximum value of the x-axis coordinate be x. max The minimum value of the x-axis coordinate is x min The maximum value of the y-axis coordinate is y max The minimum value of the y-axis coordinate is y min Then we can obtain the target bounding rectangle of the object point cloud, that is, the vertex coordinates of the minimum bounding rectangle are: (x min y min ), (x min y max ), (x max y max ), (x max y min ).

[0171] Figure 2 A schematic diagram of a target enclosing rectangle according to an embodiment of this application is shown, such as... Figure 2 As shown, a rectangle is defined by the above four vertices. This rectangle is the minimum bounding rectangle of the object point cloud, which is also the target bounding rectangle.

[0172] This application determines the coordinate orientation of an object in the robot's visual coordinate system by using the minimum bounding rectangle of the object's point cloud, without relying on the object's engineering drawing model, thus improving the convenience of obtaining the object's orientation.

[0173] Based on any of the above embodiments, determining the ratio of the length of the first side to the length of the second side includes: determining the length of the first side and the length of the second side based on the vertex coordinates; and determining the ratio based on the length of the first side and the length of the second side.

[0174] In this embodiment, the vertex coordinates of the target bounding rectangle, i.e., the minimum bounding rectangle of the object point cloud, are respectively: (x min y min ), (x min y max ), (x max y max ), (x max y min ).

[0175] After obtaining the vertex coordinates, calculate (x max -x min ) Get the length of the first side, calculate (y max -y min The length of the second side is obtained. It is understood that the larger of the lengths of the first and second sides is the length of the longer side of the target rectangle, and the smaller of the lengths of the first and second sides is the length of the shorter side of the target rectangle.

[0176] After obtaining the lengths of the first and second sides, the ratio between the two is calculated to obtain the aspect ratio of the rectangle enclosing the target. By combining the orientation data of objects with different aspect ratios, the true orientation of the object is determined, thereby improving the robot's pathfinding performance.

[0177] Based on any of the above embodiments, the determining module is specifically used for: dividing the target enclosing rectangle into multiple sub-rectangles, the sum of the areas of the multiple sub-rectangles being equal to the area of ​​the target enclosing rectangle; determining the average value of the z-axis coordinates of each point within every two adjacent sub-rectangles; determining the head coordinates and tail coordinates of the target object based on the average value and the ratio; and determining the coordinate direction of the target object based on the head coordinates and tail coordinates.

[0178] In the embodiments of this application, based on prior knowledge, the orientation of some common indoor objects is strongly related to their shape. For example, the head of a sofa, which is the backrest, is significantly higher than the seat. Similarly, the head of a bed is significantly higher than the foot of the bed.

[0179] Therefore, in practical application scenarios, such as giving a TV remote control to someone sitting on a sofa, the orientation of the object can be determined based on the height of each part of the sofa, that is, the z-axis coordinate of each point.

[0180] Specifically, the minimum bounding rectangle of the target object, i.e., the target bounding rectangle, is divided into multiple sub-rectangles of equal area. After obtaining multiple sub-rectangles, the z-axis coordinates of each point within the area of ​​each sub-rectangle are determined, and the average value of these z-axis coordinates is calculated. This average value can intuitively reflect the height of the target object within a sub-rectangle area.

[0181] The height of the target object on a certain side can be determined by averaging the z-axis coordinates within two adjacent sub-rectangles. Therefore, after obtaining the average z-axis coordinates of each point in every two adjacent sub-matrices, the object orientation, i.e., the coordinate direction of the target object in the robot's vision coordinate system, can be determined by the aspect ratio of the target object (i.e., the ratio of the length of the first side to the length of the second side) and the height of the target object on each side (i.e., the first and second sides).

[0182] Taking a sofa as an example, when it is determined that one long side of the target object is significantly higher than the other long side, the longer side with the higher height is determined as the head of the sofa, and the longer side with the lower height is determined as the tail of the sofa. Thus, the orientation of the sofa, that is, the coordinate direction of the sofa, is determined according to the direction of the head and tail vectors.

[0183] This application determines the orientation, or coordinate direction, of the target object by dividing the minimum bounding rectangle of the target object into multiple adjacent sub-rectangles and combining the aspect ratio of the minimum bounding rectangle. This process does not rely on the engineering drawing model of the object and can determine the object's orientation more quickly.

[0184] Based on any of the above embodiments, the determining module is specifically used to: determine a sub-rectangle using the intersection of the first side and the second side, the midpoint of the first side, the midpoint of the second side, and the center point of the target enclosing rectangle as vertices.

[0185] In the embodiments of this application, the target bounding rectangle, that is, the minimum bounding rectangle of the target object, includes an intersecting first side and a second side, wherein the intersection point of the first side and the second side is a vertex of the minimum bounding rectangle.

[0186] Taking the first side as the long side of the target enclosing rectangle and the second side as the wide side of the target enclosing rectangle as an example, the first line segment is determined by taking the midpoint of the two long sides of the target enclosing rectangle as the endpoint, and the second line segment is determined by taking the midpoint of the two wide sides of the target enclosing rectangle as the endpoint. The intersection of the first line segment and the second line segment is the midpoint of the target enclosing rectangle.

[0187] Figure 3 A schematic diagram of a sub-rectangle according to an embodiment of this application is shown, such as... Figure 3 As shown, a sub-rectangle is defined using one vertex of the target-enclosing rectangle, the midpoint of the adjacent long and short sides, and the midpoint of the target-enclosing rectangle as vertices. This results in four sub-rectangles of equal area, effectively dividing the target-enclosing rectangle into four equal sub-rectangles. Each of these four sub-rectangles has an adjacent sub-rectangle on the short side and a adjacent sub-rectangle on the long side of the target-enclosing rectangle.

[0188] Therefore, by using the average z-axis coordinates of each point in every two adjacent sub-rectangles, the height of the target object on the width and length sides can be determined, and thus the orientation of the target object can be identified.

[0189] This application divides the target's enclosing rectangle into four sub-rectangles of equal area, which are adjacent on both the width and length sides. The coordinate direction of the target object is determined by the average value of the z-axis coordinates of these sub-rectangles, thus achieving a quick and convenient method for determining the object's orientation.

[0190] Based on any of the above embodiments, the determining module is specifically used for: determining the adjacent first sub-rectangle and second sub-rectangle corresponding to the maximum value of the average value when the ratio is less than a preset threshold; determining the head coordinates based on the coordinates of the first intersection point of the common side of the first sub-rectangle and the second sub-rectangle and the side of the target enclosing rectangle; and determining the tail coordinates based on the coordinates of the midpoint of the side opposite the side where the first intersection point is located in the side of the target enclosing rectangle.

[0191] In this embodiment, a preset threshold, which is used to determine the shape of the target object, is a constant less than 1 and greater than 0 in some implementations. If the ratio of the length of the first side of the target object to the length of the second side of the target object, that is, the aspect ratio of the target object, is less than the preset threshold, it indicates that the shape of the target object is relatively square, and the long side and the wide side of the minimum enclosing rectangle of the target object are relatively close. At this time, based on prior knowledge, all four sides of the minimum enclosing rectangle of the target object may be the "head" or "tail" of the target object.

[0192] Meanwhile, the target enclosing rectangle is divided into 4 sub-rectangles. These 4 sub-rectangles consist of 2 adjacent sub-rectangles on the long side of the target enclosing rectangle and 2 adjacent sub-rectangles on the wide side of the target enclosing rectangle, for a total of 4 sets of adjacent sub-rectangles.

[0193] Among these four groups of adjacent sub-rectangles, the group with the highest average z-axis coordinate value is determined. This group of adjacent sub-rectangles specifically includes the first sub-rectangle and the second sub-rectangle. The first intersection point of the common side of the first and second sub-rectangles with its corresponding long or wide side is defined as the "head point" of the target object, and its coordinates are recorded as the head coordinates of the target object. The midpoint of the side opposite the first intersection point among the four sides of the target's enclosing rectangle is defined as the "tail point" of the target object, and its coordinates are recorded as the tail coordinates of the target object.

[0194] After obtaining the head and tail coordinates, let the head coordinates be (x1, y1) and the tail coordinates be (x2, y2). Then, the coordinate direction of the target object can be calculated using the following formula:

[0195]

[0196] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0197] This application embodiment determines the head and tail coordinates of the target object, determines the object's coordinate direction based on the head and tail coordinates, and then determines the object's orientation, thereby improving the pathfinding performance based on robot vision.

[0198] Based on any of the above embodiments, the determining module is further configured to: determine the third side of the target enclosing rectangle according to the object type of the target object when the ratio is greater than or equal to a preset threshold; determine the adjacent third and fourth sub-rectangles corresponding to the maximum value of the average value, wherein the third and fourth sub-rectangles are adjacent on the third side; determine the head coordinates according to the coordinates of the second intersection point of the common side of the third and fourth sub-rectangles and the third side; and determine the tail coordinates according to the coordinates of the midpoint of the side opposite the side where the second intersection point is located in the target enclosing rectangle.

[0199] In this application embodiment, a preset threshold, which is a threshold used to determine the shape of the target object, is, in some embodiments, a constant less than 1 and greater than 0. If the ratio of the length of the first side of the target object to the length of the second side of the target object, that is, the aspect ratio of the target object, is greater than or equal to the preset threshold, it indicates that the difference between the length of the long side and the length of the wide side of the minimum enclosing rectangle of the target object is large.

[0200] Based on prior knowledge, for scenarios where the target object is a sofa, the two longest sides of the minimum bounding rectangle of the target object are more likely to be the "head" or "tail" of the target object. For scenarios where the target object is a single bed, the two shortest sides of the minimum bounding rectangle of the target object are more likely to be the "head" or "tail" of the target object.

[0201] At this point, based on the object type of the target object, it is determined that the head and tail of this type of object are generally on the long side or the wide side. Taking the target object as a sofa as an example, the head of the sofa generally refers to the back of the sofa, and the back of the sofa is generally on the long side of the minimum bounding rectangle of the object point cloud of the sofa. Therefore, the long side of the target bounding rectangle can be recorded as the third side, and in the four sub-rectangles divided by the minimum bounding rectangle, two sets of two adjacent sub-rectangles on the long side, that is, the third side, are determined.

[0202] It is understandable that if the target object is a single bed, and the headboard and footboard of the single bed are on the wide side of the minimum bounding rectangle of the object's point cloud, then the wide side of the target's bounding rectangle is denoted as the third side.

[0203] Among the two adjacent sub-rectangles on the third side, determine the pair of adjacent sub-rectangles with the highest average z-axis coordinate values. This pair of adjacent sub-rectangles specifically includes the third and fourth sub-rectangles. The third and fourth sub-rectangles are adjacent on the third side. The second intersection point of the common side of the third and fourth sub-rectangles with the third side is determined as the "head point" of the target object, and the coordinates of this second intersection point are taken as the head coordinates of the target object. The midpoint of the side opposite the second intersection point among the four sides of the target's enclosing rectangle is recorded as the "tail point" of the target object, and its coordinates are taken as the tail coordinates of the target object.

[0204] After obtaining the head and tail coordinates, let the head coordinates be (x1, y1) and the tail coordinates be (x2, y2). Then, the coordinate direction of the target object can be calculated using the following formula:

[0205]

[0206] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0207] This application embodiment determines the head and tail coordinates of the target object, determines the object's coordinate direction based on the head and tail coordinates, and then determines the object's orientation, thereby improving the pathfinding performance based on robot vision.

[0208] Based on any of the above embodiments, the first side includes a first long side and a second long side, and the second side includes a first wide side and a second wide side;

[0209] The determination module is specifically used for: determining a first difference between a first average value and a second average value, wherein the first average value is the average of the z-coordinates of points within two adjacent sub-rectangles on the first long side, and the second average value is the average of the z-coordinates of points within two adjacent sub-rectangles on the second long side; determining a second difference between a third average value and a fourth average value, wherein the third average value is the average of the z-coordinates of points within two adjacent sub-rectangles on the first wide side, and the fourth average value is the average of the z-coordinates of points within two adjacent sub-rectangles on the second wide side; if the first difference is greater than the second difference, determining the coordinates of the third intersection point of the common edge of the two sub-rectangles corresponding to the larger of the first and second average values ​​with the edge of the target enclosing rectangle as the head coordinates; determining the tail coordinates based on the midpoint coordinates of the side opposite the edge containing the third intersection point of the target enclosing rectangle; if the first difference is less than the second difference, determining the coordinates of the fourth intersection point of the common edge of the two sub-rectangles corresponding to the larger of the third and fourth average values ​​with the edge of the target enclosing rectangle as the head coordinates; and determining the tail coordinates based on the midpoint coordinates of the side opposite the edge containing the fourth intersection point of the target enclosing rectangle.

[0210] In this embodiment, the four sides of the target enclosing rectangle are specifically a first long side, a first wide side, a second long side, and a second wide side. The target enclosing rectangle is divided into four sub-rectangles of equal area. Among the four sub-rectangles, there are two sets of two adjacent sub-rectangles on the long side of the target enclosing rectangle, and two sets of two adjacent sub-rectangles on the wide side of the target enclosing rectangle.

[0211] Specifically, the average z-axis coordinates of each point within two adjacent sub-rectangles on the long side of the target's enclosing rectangle are denoted as the first average and the second average. The average z-axis coordinates of each point within two adjacent sub-rectangles on the wide side of the target's enclosing rectangle are denoted as the third average and the fourth average.

[0212] Calculate the difference between the first average and the second average, and denote it as the first difference. Calculate the difference between the third average and the fourth average, and denote it as the second difference.

[0213] If the first difference is greater than the second difference, it indicates that the height difference between the two longer sides of the target object is greater. If the first difference is less than the second difference, it indicates that the height difference between the two shorter sides of the target object is greater.

[0214] Based on prior knowledge, for common indoor objects such as sofas and beds, the height difference between the head and tail is generally significantly greater than the height difference between the two sides. For example, the two armrests of a sofa are the same height, and the two sides of a bed are also the same height.

[0215] Therefore, if the first difference is greater than the second difference, then among the two groups of two sub-rectangles that are adjacent on the long side, the group of two sub-rectangles with the larger average value of the z-axis coordinates is determined. The third intersection point of the common side of these two sub-rectangles with the long side of the target enclosing rectangle is determined as the head point, and the coordinates of the third intersection point are taken as the head coordinates. The midpoint of the side opposite the side where the third intersection point is located among the four sides of the target enclosing rectangle is determined as the tail point, and its coordinates are taken as the tail coordinates.

[0216] If the first difference is less than the second difference, then among the two groups of two adjacent sub-rectangles on the wide side, determine the group of two sub-rectangles with the larger average z-axis coordinate value. The fourth intersection point of the common side of these two sub-rectangles with the wide side of the target enclosing rectangle is determined as the head point, and the coordinates of the fourth intersection point are taken as the head coordinates. The midpoint of the side opposite the fourth intersection point among the four sides of the target enclosing rectangle is determined as the tail point, and its coordinates are taken as the tail coordinates.

[0217] After obtaining the head and tail coordinates, let the head coordinates be (x1, y1) and the tail coordinates be (x2, y2). Then, the coordinate direction of the target object can be calculated using the following formula:

[0218]

[0219] Where θ is the orientation angle, (x1, y1) are the head coordinates, and (x2, y2) are the tail coordinates.

[0220] This application embodiment determines the head and tail coordinates of the target object, determines the object's coordinate direction based on the head and tail coordinates, and then determines the object's orientation, thereby improving the pathfinding performance based on robot vision.

[0221] Based on any of the above embodiments, the determining module is specifically used to determine the N largest z-coordinate values ​​in each of two adjacent sub-rectangles, where N is a positive integer; and to determine the average value based on the N z-coordinate values.

[0222] In this embodiment of the application, when determining the average value of the z-axis coordinates of each point within two adjacent sub-rectangles, the N largest z-coordinate values ​​can be determined from all points in the region of these two adjacent sub-rectangles. Specifically, all points in this region can be sorted according to the order of their z-axis coordinates, and the first N points after sorting can be retained. The average value can be determined based on the z-axis coordinates of these N points.

[0223] It is understandable that if the number of points in the region is less than N, the average value is determined based on the z-axis coordinates of all points in the region.

[0224] This application calculates the average value of the N largest z-axis coordinates within a region to describe the height of the target object within that region. This more accurately reflects the actual height of the object and improves the accuracy of determining the object's orientation, i.e., the coordinate direction of the target object.

[0225] In some embodiments of the present invention, a device for determining the coordinate orientation of an object is provided. Figure 5 A structural block diagram of an object coordinate orientation determination device according to an embodiment of this application is shown, such as... Figure 5 As shown, the object coordinate orientation determination device 500 includes: a memory 502 for storing programs or instructions; and a processor 504 for executing the program or instructions to implement the steps of the object coordinate orientation determination method provided in any of the above embodiments. Therefore, the object coordinate orientation determination device also includes all the beneficial effects of the object coordinate orientation determination method provided in any of the above embodiments. To avoid repetition, these effects will not be described again here.

[0226] In some embodiments of the present invention, a readable storage medium is provided on which a program or instructions are stored. When the program or instructions are executed by a processor, they implement the steps of the method for determining the coordinate orientation of an object as provided in any of the above embodiments. Therefore, the readable storage medium also includes all the beneficial effects of the method for determining the coordinate orientation of an object as provided in any of the above embodiments. To avoid repetition, these effects will not be described again here.

[0227] In some embodiments of the present invention, a computer program product is provided. The computer program product is stored in a storage medium. When the computer program product is executed by at least one processor, it implements the steps of the method for determining the coordinate orientation of an object as provided in any of the above embodiments. Therefore, the computer program product also includes all the beneficial effects of the method for determining the coordinate orientation of an object as provided in any of the above embodiments. To avoid repetition, these effects will not be described again here.

[0228] In some embodiments of the present invention, an electronic device is provided, including a means for determining the coordinate direction of an object as provided in any of the above embodiments; and / or a readable storage medium as provided in any of the above embodiments; and / or a computer program product as provided in any of the above embodiments, thus also including the means for determining the coordinate direction of an object as provided in any of the above embodiments; and / or a readable storage medium as provided in any of the above embodiments; and / or all the beneficial effects of the computer program product as provided in any of the above embodiments will not be repeated here to avoid repetition.

[0229] In some embodiments of the present invention, a robot is provided, including a device for determining the coordinate direction of an object as provided in any of the above embodiments; and / or a readable storage medium as provided in any of the above embodiments; and / or a computer program product as provided in any of the above embodiments, thus also including the device for determining the coordinate direction of an object as provided in any of the above embodiments; and / or a readable storage medium as provided in any of the above embodiments; and / or all the beneficial effects of the computer program product as provided in any of the above embodiments will not be repeated here to avoid repetition.

[0230] Based on the above embodiments, the robot further includes: a navigation module, which, upon receiving a walking instruction to walk to a target object, determines the target navigation coordinates according to the coordinate direction of the target object and controls the robot to walk to the target navigation coordinates.

[0231] In this embodiment, the robot includes a navigation module for semantic navigation. For example, a first user places their phone on the robot and commands it to hand the phone to a second user sitting on a sofa. The robot, based on the recognized coordinates of the sofa, determines the rear of the sofa (i.e., one side of the seat cushion) as the target navigation coordinate and moves to that coordinate, thus accurately handing the phone to the second user. This avoids the awkward situation where the robot travels to the side or back of the sofa, requiring the second user to get up to retrieve the phone, and improves the robot's pathfinding performance.

[0232] In the description of this application, the term "multiple" refers to two or more. Unless otherwise expressly defined, the terms "upper," "lower," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. The terms "connection," "installation," "fixing," etc., should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection or an indirect connection through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0233] In the description of this application, the terms "one embodiment," "some embodiments," "specific embodiment," etc., refer to a specific feature, structure, material, or characteristic described in connection with that embodiment or example, which is included in at least one embodiment or example of this application. In this application, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0234] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for determining the coordinate orientation of an object, characterized in that, include: Based on the object point cloud of the target object, a target bounding rectangle of the object point cloud is determined, and the target bounding rectangle includes an intersecting first side and a second side. Determine the ratio of the length of the first side to the length of the second side; The coordinate direction of the target object is determined based on the coordinate values ​​of each point within the target enclosure rectangle and the ratio value. The coordinate values ​​include x-axis coordinate values, y-axis coordinate values, and z-axis coordinate values; Determining the coordinate direction of the target object based on the coordinate values ​​of each point within the target's enclosing rectangle and the ratio includes: The target enclosure rectangle is divided into multiple sub-rectangles, and the sum of the areas of the multiple sub-rectangles is equal to the area of ​​the target enclosure rectangle; Determine the average value of the z-axis coordinates of each point within every two adjacent sub-rectangles; The head and tail coordinates of the target object are determined based on the average value and the ratio. The coordinate direction of the target object is determined based on the head coordinates and the tail coordinates.

2. The method according to claim 1, characterized in that, The step of determining the target bounding rectangle of the object point cloud based on the object point cloud of the target object includes: Obtain the maximum value of the x-axis coordinate and the maximum value of the y-axis coordinate for each point within the point cloud of the object; The vertex coordinates of the target enclosing rectangle are determined based on the maximum value of the x-axis coordinate and the maximum value of the y-axis coordinate. The target enclosing rectangle is determined based on the vertex coordinates.

3. The method according to claim 2, characterized in that, Determining the ratio of the length of the first side to the length of the second side includes: Based on the vertex coordinates, determine the length of the first side and the length of the second side; The ratio is determined based on the lengths of the first side and the second side.

4. The method according to claim 1, characterized in that, The step of dividing the target-enclosing rectangle into multiple adjacent sub-rectangles includes: A sub-rectangle is determined by taking the intersection of the first side and the second side, the midpoint of the first side, the midpoint of the second side, and the center point of the target enclosing rectangle as vertices.

5. The method according to claim 3, characterized in that, Determining the head and tail coordinates of the target object based on the average value and the ratio includes: If the ratio is less than a preset threshold, determine the adjacent first sub-rectangle and second sub-rectangle corresponding to the maximum value of the average value; The head coordinates are determined based on the coordinates of the first intersection point of the common side of the first sub-rectangle and the second sub-rectangle and the side of the target enclosing rectangle; The tail coordinates are determined based on the coordinates of the midpoint of the side opposite the side containing the first intersection point in the rectangle enclosing the target.

6. The method according to claim 5, characterized in that, Also includes: If the ratio is greater than or equal to the preset threshold, the third side of the target enclosing rectangle is determined according to the object type of the target object; Determine the adjacent third and fourth sub-rectangles corresponding to the maximum value of the average value, wherein the third and fourth sub-rectangles are adjacent on the third side; The head coordinates are determined based on the coordinates of the second intersection point of the common side of the third sub-rectangle and the fourth sub-rectangle and the third side; The tail coordinates are determined based on the coordinates of the midpoint of the side opposite the side containing the second intersection point in the rectangle surrounding the target.

7. The method according to claim 3, characterized in that, Before determining the third side of the target bounding rectangle based on the object type of the target object, the method further includes: Acquire image data of the target object; The object type is determined based on the image data; Determining the third side of the target bounding rectangle based on the object type of the target object includes: Based on the object type, the third side is determined from a preset database, wherein the third side is either the longer or shorter side of the rectangle enclosing the target; or Display the image data and the target bounding rectangle; Receive selection input for the third side of the target enclosing rectangle; In response to the selection input, the third side is determined.

8. The method according to claim 3, characterized in that, The first side includes a first long side and a second long side, and the second side includes a first wide side and a second wide side; Determining the head and tail coordinates of the target object based on the average value and the ratio includes: Determine a first difference between a first average value and a second average value, wherein the first average value is the average of the z-coordinates of points within two adjacent sub-rectangles on the first long side, and the second average value is the average of the z-coordinates of points within two adjacent sub-rectangles on the second long side. Determine a second difference between the third average and the fourth average, wherein the third average is the average of the z-coordinates of points within two adjacent sub-rectangles on the first wide side, and the fourth average is the average of the z-coordinates of points within two adjacent sub-rectangles on the second wide side. If the first difference is greater than the second difference, the coordinates of the third intersection point of the common side of the two sub-rectangles corresponding to the larger of the first average and the second average with the side of the target enclosing rectangle are determined as the head coordinates; the tail coordinates are determined according to the coordinates of the midpoint of the side opposite to the side where the third intersection point is located in the side of the target enclosing rectangle. If the first difference is less than the second difference, the coordinates of the fourth intersection point of the common side of the two sub-rectangles corresponding to the larger of the third average and the fourth average with the side of the target enclosing rectangle are determined as the head coordinates; the tail coordinates are determined according to the coordinates of the midpoint of the side opposite the side where the fourth intersection point is located in the side of the target enclosing rectangle.

9. The method according to claim 3, characterized in that, Determining the coordinate direction of the target object based on the head coordinates and the tail coordinates includes: The orientation angle of the target object is determined based on the head coordinates and the tail coordinates; The coordinate direction is determined based on the orientation angle.

10. The method according to any one of claims 4 to 8, characterized in that, The maximum value of the z-axis coordinate of the intersection point of the first straight line and the object point cloud is greater than the maximum value of the z-axis coordinate of the intersection point of the second straight line and the object point cloud. Wherein, the first straight line is a straight line that passes through the head coordinate and is perpendicular to both the x-axis and the y-axis, and the second straight line is a straight line that passes through the tail coordinate and is perpendicular to both the x-axis and the y-axis.

11. The method according to any one of claims 4 to 8, characterized in that, The determination of the average z-axis coordinate value of each point within every two adjacent sub-rectangles includes: In each of the two adjacent sub-rectangles, determine the N z-coordinate values ​​with the largest coordinate values, where N is a positive integer; The average value is determined based on the N z-coordinate values.

12. A device for determining the coordinate orientation of an object, characterized in that, include: Memory, used to store programs or instructions; A processor for implementing the steps of the determining method as described in any one of claims 1 to 11 when executing the program or instructions.

13. A readable storage medium having a program or instructions stored thereon, characterized in that, When the program or instructions are executed by the processor, they implement the steps of the determination method as described in any one of claims 1 to 11.

14. A computer program product, said computer program product being stored in a storage medium, characterized in that, When the computer program product is executed by at least one processor, it implements the steps of the determination method as described in any one of claims 1 to 11.

15. An electronic device, characterized in that, include: The apparatus for determining the coordinate orientation of an object as described in claim 12; and / or The readable storage medium as described in claim 13; and / or The computer program product as described in claim 14.

16. A robot, characterized in that, include: The apparatus for determining the coordinate orientation of an object as described in claim 12; and / or The readable storage medium as described in claim 13; and / or The computer program product as described in claim 14.

17. The robot according to claim 16, characterized in that, Also includes: The navigation module is used to determine the target navigation coordinates based on the coordinate direction of the target object when it receives a walking instruction to walk to the target object, and control the robot to walk to the target navigation coordinates.

Images