Object localization system and method for object localization

Abstract

An object positioning system is provided, including a processing device, a sensing camera, and a memory. The sensing camera is coupled to the processing device and mounted on a self-propelled device to generate image frames. The processing device executes computer-readable code stored in the memory to: generate a mask of an entity in the image frame and determine its category using an instance segmentation model; project the mask onto a bird's-eye view plane in a global coordinate system to generate a projection mask; identify the front edge of the projection mask relative to the sensing camera; determine a reference position corresponding to the front edge; and generate a measured position of the entity on the bird's-eye view plane based on the reference position and the entity category.

Claims

1. An object positioning system, comprising: A processing device; A sensing camera, coupled to the processing device and mounted on a self-propelled device, wherein the sensing camera is configured to generate an image frame; and a memory, including computer-readable code, executed by the processing device to: generate a mask of an entity in the image frame and determine a category of the entity using an instance segmentation model; project the mask onto a bird's-eye view (BEV) plane of a global coordinate system associated with the self-propelled device to generate a projected mask; identify a front edge of the projected mask on the bird's-eye view plane relative to the sensing camera; determine a reference position corresponding to the front edge, wherein the reference position includes at least one set of coordinates representing the entity on the bird's-eye view plane; and generate a measured position of the entity on the bird's-eye view plane based on the reference position and the category of the entity.

2. The object positioning system as described in claim 1, wherein the memory stores a camera pose of the sensing camera associated with the self-propelled device, and the computer-readable code is executed by the processing device to determine a spatial transformation from a camera coordinate system of the sensing camera to the global coordinate system based on the camera pose, and to define the bird's-eye view plane based on the spatial transformation.

3. The object positioning system as described in claim 2, wherein the computer-readable code is executed by the processing device to: project a plurality of projection lines from a reference point on the bird's-eye view plane onto the bird's-eye view plane of the global coordinate system, based on the camera orientation.

4. The object positioning system as described in claim 3, wherein the reference point on the bird's-eye view plane corresponds to a projected position of the sensing camera on the bird's-eye view plane.

5. The object positioning system as described in claim 2, wherein the camera pose includes external and internal parameters of the sensing camera, and wherein the external parameters include height information, a horizontal position, and orientation information relative to the global coordinate system.

6. The object positioning system as claimed in claim 1, wherein the computer-readable code is executed by the processing device to identify the front edge by capturing a boundary outline projected onto the bird's-eye view plane.

7. The object positioning system as claimed in claim 6, wherein the computer-readable code is executed by the processing device to: identify a front group of pixels of the boundary contour of the projection mask, wherein the front group of pixels is located on the side of the projection mask facing a reference point on the bird's-eye view plane; select an optimal rectangle from a group of candidate rectangles suitable for surrounding the front group of pixels, based on the respective distances between the front group of pixels and the group of candidate rectangles; adjust the optimal rectangle according to the category of the entity to obtain an adjusted rectangle representing the entity on the bird's-eye view plane; and generate the reference position based on the adjusted rectangle.

8. The object positioning system as described in claim 7, wherein the operation of selecting the optimal rectangle from the candidate rectangle group further includes: A convex hull is generated based on the front pixel group; a front edge of the convex hull is identified, wherein the front edge is located on the side of the convex hull facing the reference point on the bird's-eye view plane; and the best rectangle suitable for enclosing the convex hull is determined from the candidate rectangle group based on the distance between the candidate rectangle group and the front edge.

9. The object positioning system as described in claim 1, further comprising: Another sensing camera is configured to synchronously generate a first wide-angle image and a second wide-angle image with the sensing camera. The first wide-angle image and the second wide-angle image have an overlapping field of view. The computer-readable code is executed by the processing device to: generate a first mask and a second mask for an entity in the first wide-angle image and the second wide-angle image, respectively, and identify a category of the entity using the instance segmentation model; project the first mask and the second mask onto the bird's-eye view plane of the global coordinate system associated with the self-propelled device to generate a first projection mask and a second projection mask; and identify a first front edge of the first projection mask relative to the sensing camera and a second front edge of the second projection mask relative to the other sensing camera on the bird's-eye view plane. Determine a first reference position corresponding to one of the first front edges and a second reference position corresponding to one of the second front edges, wherein each of the first reference position and the second reference position includes at least one set of coordinates representing the entity on the bird's-eye view plane; and merge the first reference position and the second reference position when the first reference position and the second reference position satisfy a first predetermined condition to generate the measured position of the entity at a current timestamp.

10. The object positioning system as claimed in claim 9, wherein the first predetermined condition includes the first reference position and the second reference position being within a first predetermined distance on the bird's-eye view plane.

11. The object positioning system of claim 1, wherein the memory further stores a historical trajectory including a previous position of the entity in the bird's-eye view plane, and the computer-readable code is executed by the processing device to: generate a predicted position in the bird's-eye view plane based on the historical trajectory; calculate a distance between the measured position and the predicted position; and, in response to the distance between the measured position and the predicted position satisfying a second predetermined condition, connect the measured position and the predicted position to obtain an updated position of the entity in the bird's-eye view plane.

12. The object positioning system as claimed in claim 11, wherein the second predetermined condition includes the distance between the measured position and the predicted position being within a second predetermined distance on the bird's-eye view plane.

13. The object positioning system as described in claim 1, wherein the bird's-eye view plane is defined as a ground plane of the global coordinate system.

14. The object positioning system as described in claim 1, wherein the sensing camera is a fisheye camera.

15. An object positioning method, executed by a processing device, the object positioning method comprising: An image frame is generated by a sensing camera mounted on a self-propelled device; A mask of an object in the image frame is generated, and a category of the object is identified using an instance segmentation model; the mask is projected onto a bird's-eye view plane of a global coordinate system associated with the self-propelled device to generate a projection mask; on the bird's-eye view plane, a front edge of the projection mask relative to the sensing camera is identified; a reference position corresponding to the front edge is determined, wherein the reference position includes at least one set of coordinates representing the object in the bird's-eye view plane; and based on the reference position and the category of the object, a measured position of the object in the bird's-eye view plane is generated.

16. The object positioning method as described in claim 15, wherein the operation of projecting the mask onto the bird's-eye view plane of the global coordinate system further includes: Based on a camera pose, a spatial transformation from one of the camera coordinate systems of the sensing camera to the global coordinate system is determined, and the bird's-eye view plane is defined according to the spatial transformation.

17. The object positioning method as described in claim 16, further comprising: Based on the camera orientation, multiple projection lines are extended from one of the reference points of the bird's-eye view plane to project the mask onto the bird's-eye view plane of the global coordinate system.

18. The object positioning method as described in claim 17, wherein the reference point on the bird's-eye view plane corresponds to a projected position of the sensing camera on the bird's-eye view plane.

19. The object positioning method as described in claim 16, wherein the camera pose includes extrinsic and intrinsic parameters of the sensing camera, and wherein the extrinsic parameters include height information, a horizontal position, and orientation information of the sensing camera relative to the global coordinate system.

20. The object positioning method as described in claim 15, wherein the operation of identifying one of the front edges of the projection mask relative to the sensing camera on the bird's-eye view plane further includes: The front edge is identified by capturing the boundary outline of the projection masked on the bird's-eye view plane.

21. The object positioning method as claimed in claim 20, wherein the operation of determining the reference position corresponding to the front edge further comprises: identifying a front pixel group of the boundary contour of the projection mask, wherein the front pixel group is located on the side of the projection mask facing a reference point on the bird's-eye view plane; selecting an optimal rectangle from a candidate rectangle group suitable for surrounding the front pixel group based on the respective distances between the front pixel group and the candidate rectangle group; adjusting the size of the optimal rectangle based on the category of the object to obtain an adjusted rectangle for representing the object on the bird's-eye view plane; and generating the reference position based on the adjusted rectangle.

22. The object positioning method as described in claim 21, wherein the operation of selecting the best rectangle from the candidate rectangle group further includes: A convex hull is generated based on the front pixel group; a front edge of the convex hull is identified, wherein the front edge is located on the side of the convex hull facing the reference point on the bird's-eye view plane; and the best rectangle suitable for enclosing the convex hull is determined from the candidate rectangle group based on the respective distances between the candidate rectangle group and the front edge.

23. The object positioning method as described in claim 15, further comprising: A first wide-angle image and a second wide-angle image are generated simultaneously by the sensing camera and another sensing camera, and the first wide-angle image and the second wide-angle image have an overlapping field of view; using the instance segmentation model, a first mask and a second mask of an entity are generated in the first wide-angle image and the second wide-angle image respectively, and the category of the object is determined; the first mask and the second mask are projected onto the bird's-eye view plane of the global coordinate system associated with the self-propelled device to generate a first projection mask and a second projection mask; on the bird's-eye view plane, a first front edge of the first projection mask is identified relative to the sensing camera, and a second front edge of the second projection mask is identified relative to the other sensing camera; Determine a first reference position corresponding to one of the first front edges and a second reference position corresponding to one of the second front edges, wherein each of the first reference position and the second reference position includes at least one set of coordinates representing the bird's-eye view plane; and merge the first reference position and the second reference position when the first reference position and the second reference position satisfy a first predetermined condition to generate the measured position of the entity at a current timestamp.

24. The object positioning method as described in claim 23, wherein the first predetermined condition includes the first reference position and the second reference position being within a first predetermined distance on the bird's-eye view plane.

25. The object positioning method as described in claim 15, further comprising: Based on the historical trajectory of one of the objects, a predicted location is generated on the bird's-eye view plane; Calculate the distance between the measured location and the predicted location; And when the distance between the measured position and the predicted position meets a second predetermined condition, the measured position and the predicted position are linked to obtain an updated position of the object on one of the bird's-eye view planes.

26. The object positioning method as described in claim 25, wherein the second predetermined condition includes the distance between the measured position and the predicted position being within a second predetermined distance on the bird's-eye view plane.

Images