Target object recognition device, method for recognizing target object, and program

By estimating the floor surface and setting a detection area parallel to it, the device reduces calculation time for object recognition, improving real-time performance.

JP2024168758APending Publication Date: 2024-12-05SINTOKOGIO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023085684
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Existing object recognition devices that utilize 3D point cloud information require significant calculation time for object recognition, which hinders real-time performance.

Method used

The device employs an imaging unit to capture a three-dimensional point cloud, estimates the floor surface, sets a detection area parallel to this surface, and performs object detection within this limited area using processors to reduce calculation time.

Benefits of technology

This approach significantly reduces the time required for object recognition by limiting the three-dimensional point group used for detection, enhancing real-time performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024168758000001_ABST
    Figure 2024168758000001_ABST
Patent Text Reader

Abstract

To reduce an operation time required to recognize a target object.SOLUTION: A target object recognition device (1) includes: an imaging unit (20); and at least one processor. The at least one processor acquires the three-dimensional point group of an object and estimates a floor surface (31) on which a target object (W) is arranged on the basis of the acquired three-dimensional point group. The at least one processor sets a columnar region of which floor surface is parallel to the floor surface (31) as a detection region (41) for detecting the target object (W).SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an object recognition device, an object recognition method, and a program. [Background technology]

[0002] Patent Document 1 discloses an information processing device that detects a target object from an image acquired by an imaging unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-28196 Summary of the Invention [Problem to be solved by the invention]

[0004] In an object recognition device that acquires three-dimensional point cloud information from images captured by an imaging unit and recognizes objects from the three-dimensional point cloud information, there is a demand to shorten the calculation time required to recognize objects in order to improve real-time performance.

[0005] One aspect of the present disclosure aims to reduce the calculation time required to recognize an object. [Means for solving the problem]

[0006] In order to solve the above problems, an object recognition device according to an embodiment of the present disclosure recognizes an object. The object recognition device includes an imaging unit that captures an image of a subject within an imaging range, and one or more processors. The one or more processors execute a point cloud acquisition process, a floor surface estimation process, and a detection area setting process. In the point cloud acquisition process, the one or more processors control the imaging unit to acquire a three-dimensional point cloud of the subject. In the floor surface estimation process, the one or more processors estimate a floor surface on which the object is located based on the three-dimensional point cloud. In the detection area setting process, the one or more processors set a columnar area whose bottom surface is parallel to the floor surface as a detection area in which the object is detected.

[0007] The object recognition device according to each aspect of the present disclosure may be realized by a computer. In this case, the object recognition device program that causes a computer to operate as each unit (software element) of the object recognition device to realize the object recognition device, and a computer-readable recording medium on which the program is recorded, are also included in the scope of the present disclosure. Effect of the Invention

[0008] According to one aspect of the present disclosure, it is possible to reduce the calculation time required to recognize an object. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration of an object recognition device according to an embodiment of the present disclosure. [Diagram 2] 1 is a block diagram showing a configuration of an object recognition device according to an embodiment of the present disclosure. [Diagram 3] 1 is a flowchart showing a flow of an object recognition method by an object recognition device according to an embodiment of the present disclosure. [Figure 4] 4 is a diagram showing an example of a three-dimensional shape that is a target of a point cloud acquisition process in the object recognition method shown in FIG. 3. [Diagram 5]4 is a diagram showing an example of a detection area set by a detection area setting process in the object recognition method shown in FIG. 3. [Figure 6] 4 is a diagram used to explain a detection area setting process and a frame determining process in the object recognition method shown in FIG. 3. [Figure 7] 4 is a diagram used to explain a detection area setting process and a frame determining process in the object recognition method shown in FIG. 3. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Fig. 1 is a schematic diagram showing a configuration of an object recognition device according to an embodiment of the present disclosure. As shown in Fig. 1, the object recognition device 1 includes a control device 10 and an imaging unit 20. Hereinafter, for convenience of explanation, the up-down direction and the left-right direction of the object recognition device 1 are defined as shown by the arrows in Fig. 1.

[0011] The control device 10 is connected to an imaging unit 20. The imaging unit 20 is, for example, a ToF (Time of Flight) camera. The imaging unit 20 captures an image of a subject inside an imaging range 21 at a predetermined period. The imaging unit 20 also measures the distance to the subject. For example, the imaging unit 20 measures the distance to the subject by irradiating infrared light and detecting the time it takes for the infrared light to hit the subject, be reflected, and return.

[0012] The control device 10 controls the imaging unit 20 to obtain an imaged image and a distance image of a subject inside the imaging range 21. The distance image is an image of the distance from the imaging unit 20 to the subject, for example, the distance from the imaging unit 20 to the subject converted into color.

[0013] In FIG. 1, the imaging range 21 of the imaging unit 20 includes the belt 31 of the belt conveyor 30 and the workpiece W placed on the belt 31 and transported. The belt conveyor 30 transports the workpiece W in a direction perpendicular to the plane of the paper, for example, by moving the belt 31 in a direction perpendicular to the plane of the paper. The workpiece W is an example of an object, and is a target for picking by a robot arm (not shown). The belt 31 is an example of a floor surface. In FIG. 1, the belt 31 is shown as a flat plane, but it may have an inclination. In addition, the belt 31 may have unevenness that is equal to or smaller than the measurement error of the distance by the imaging unit 20.

[0014] The control device 10 sets a detection area 41 inside the imaging range 21 of the imaging unit 20 by a method described later in detail. The detection area 41 is an area for detecting an object, and is set based on the captured image and distance image acquired from the imaging unit 20. In FIG. 1, the detection area 41 is smaller than the imaging range 21 of the imaging unit 20. The control device 10 detects the position and posture of the work W placed on the belt 31 of the belt conveyor 30 from within the detection area 41. The control device 10 sets an object area 42 of the work W based on the position and posture of the work W. The object area 42 is an area indicating the position and posture of the recognized object. The detection area 41 and the object area 42 will be described later in detail. The range for detecting the position and posture of the work W is limited to the inside of the detection area 41, thereby reducing the calculation time required to recognize the work W. Note that, technically, the detection area 41 can be set to include the imaging range 21 of the imaging unit 20.

[0015] 2 is a block diagram showing a configuration of an object recognition device according to an embodiment of the present disclosure. As shown in FIG. 2, the control device 10 includes a processor 11, a primary memory 12, a secondary memory 13, and an input / output interface 14.

[0016] The processor 11 is, for example, a central processing unit (CPU). The processor 11 may be, for example, a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these.

[0017] The primary memory 12 is, for example, a semiconductor random access memory (RAM). The secondary memory 13 is an example of a storage unit, and stores the program P1. The secondary memory 13 is, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), an optical disk drive (ODD), or a combination of these.

[0018] The processor 11 loads the program P1 stored in the secondary memory 13 onto the primary memory 12. Then, the processor 11 controls each part of the object recognition device 1 according to instructions included in the program P1 loaded onto the primary memory 12.

[0019] In this embodiment, a configuration is adopted in which each part of the object recognition device 1 is controlled using a single processor (processor 11), but the present disclosure is not limited to this. That is, a configuration may be adopted in which each part of the object recognition device 1 is controlled using multiple processors. In this case, the multiple processors that cooperate to control each part of the object recognition device 1 may be provided in a single computer and configured to be able to communicate with each other, or may be provided in a distributed manner in multiple computers and configured to be able to communicate with each other via a network. As an example, a mode in which a processor built in a computer that constitutes a cloud server and a processor built in a computer owned by a user of the cloud server cooperate to execute the program P1, etc., can be considered.

[0020] An imaging unit 20 is connected to the input / output interface 14. Examples of the input / output interface 14 include interfaces such as a USB, an A / D converter, and EtherCat (Ethernet for Control Automation Technology).

[0021] 3 is a flowchart showing a flow of an object recognition method by an object recognition device according to an embodiment of the present disclosure. In S100, the processor 11 of the control device 10 executes a point cloud acquisition process. The processor 11 acquires a captured image and a distance image from the imaging unit 20. The processor 11 acquires a three-dimensional point cloud based on the acquired captured image and distance image. The three-dimensional point cloud is data indicating the three-dimensional shape of the subject of the imaging unit 20. Any known technology can be used as a method for acquiring the three-dimensional point cloud based on the captured image and distance image.

[0022] Fig. 4 is a diagram showing an example of a three-dimensional shape that is the target of the point cloud acquisition process. Fig. 4 shows the three-dimensional shape of the belt 31 of the belt conveyor 30 shown in Fig. 1 and the workpiece W placed on the belt 31. The infrared light irradiated from the imaging unit 20 shown in Fig. 1 is reflected by, for example, the belt 31 and the top surface 51 and side surface 52 of the workpiece W. The three-dimensional point cloud includes data on the distance from the imaging unit 20 shown in Fig. 1 for the point cloud on the belt 31 and the top surface 51 and side surface 52 of the workpiece W shown in Fig. 4, for example.

[0023] In S101, the processor 11 determines whether or not a detection area 41 for detecting an object has already been set. Since the detection area 41 has not yet been set at the timing when the 3D point cloud is acquired for the first time in S100 (S101: NO), the processor 11 proceeds to the process of S102.

[0024] In S102, the processor 11 executes a floor surface estimation process to estimate a floor surface on which an object is placed from the three-dimensional point cloud acquired in S100. For example, the processor 11 estimates the floor surface by plane estimation using RANSAC (Random Sample Consensus) for the three-dimensional point cloud acquired in S100.

[0025] In the next step S103, the processor 11 executes a detection area setting process to set a detection area 41 for detecting an object on the floor surface estimated in S102. The detection area 41 set by the detection area setting process is a columnar area, the bottom surface of which is parallel to the floor surface estimated in S102.

[0026] FIG. 5 is a diagram showing an example of a detection area set by the detection area setting process. The detection area 41 shown in FIG. 5 is arranged in an X1Y1Z1 coordinate system. The X1Y1Z1 coordinate system is an example of a first coordinate system, and is a virtual three-dimensional area. The center of the detection area 41 is located at the origin O1 of the X1Y1Z1 coordinate system. The X1 axis and the Y1 axis are parallel to the bottom surface 41A of the detection area 41 and are perpendicular to each other. The Z1 axis is perpendicular to the bottom surface 41A of the detection area 41. The length r of the detection area 41 in the Z1 axis direction is shorter than the distance between the imaging unit 20 shown in FIG. 1 and the floor surface estimated in S102.

[0027] In S103, the processor 11 converts the detection area 41 in the X1Y1Z1 coordinate system into the X2Y2Z2 coordinate system. The X2Y2Z2 coordinate system is an example of a second coordinate system. The Z2 axis of the X2Y2Z2 coordinate system is parallel to the normal vector of the floor surface estimated in S102. The X2 axis and the Y2 axis of the X2Y2Z2 coordinate system are parallel to the floor surface estimated in S102. Hereinafter, the unit vector of the normal vector of the floor surface estimated in S102 is referred to as unit normal vector n. n=(a, b, c) T where T stands for transpose.

[0028] An example of a formula used in the coordinate conversion process in S103 is shown in formula (1). q=R n R θ p+U …(1) In formula (1), p is a coordinate in the X1Y1Z1 coordinate system before the coordinate transformation, and is located inside the detection area 41. q is a coordinate in the X2Y2Z2 coordinate system after the coordinate transformation. R n is a unit vector e that is parallel to the Z1 axis and has its base point at the origin O1 of the X1Y1Z1 coordinate system. Z1 is a transformation matrix used in coordinate transformation for rotating the detection area 41 so that it coincides with the unit normal vector n. R θ is a transformation matrix used in coordinate transformation to rotate the detection area 41 by a rotation angle θ around the Z1 axis. n and R θAn example is shown in equations (2) and (3).

number

number

[0029] t Z is set to a value greater than 0. That is, a gap is provided between the detection area 41 and the floor surface estimated in S102. This gap is provided to distinguish between the floor surface estimated in S102 and something other than the floor surface, and is set to a larger value as the measurement error of the distance by the imaging unit 20 becomes larger.

[0030] An example of the detection area 41 after the coordinate conversion in S103 is shown in Fig. 6. The detection area 41 shown in Fig. 6 is separated in the Z2 axis direction from the belt 31, which was estimated to be the floor surface in S102. The inside of the detection area 41 shown in Fig. 6 includes the top surface 51 and the upper part of the side surface 52 of the workpiece W placed on the belt 31.

[0031] In S104 of Fig. 3, the processor 11 executes an object detection process. The processor 11 detects the position and posture of the target object based on the point cloud located inside the detection area 41 set in S103, among the three-dimensional point cloud acquired in S100. For example, as shown in Fig. 6, the processor 11 converts the three-dimensional point cloud inside the detection area 41 into a two-dimensional projected point cloud by projecting the top surface 51 and the upper part of the side surface 52 of the workpiece W located inside the detection area 41, among the point cloud of the workpiece W, onto the bottom surface 41A of the detection area 41.

[0032] FIG. 7 is a diagram showing an example of a two-dimensional projected point cloud. The two-dimensional projected point cloud shown in FIG. 7 includes a projected point cloud 61 obtained by projecting a point cloud located on the upper surface 51 of the workpiece W onto the bottom surface 41A of the detection area 41, and a projected point cloud 62 obtained by projecting a point cloud located on the upper part of the side surface 52 located inside the detection area 41 onto the bottom surface 41A of the detection area 41. The processor 11 performs clustering on the projected point cloud, and further detects the position and posture of the workpiece W. The position of the workpiece W is, for example, the position of the center of gravity of the two-dimensional projected point clouds 61 and 62. Clustering is a method of grouping data based on the similarity between data. For example, based on the Euclidean distance between the point clouds, point clouds closer than a distance set as a criterion are regarded as being in the same group, and grouping is performed. The posture of the workpiece W includes, for example, the orientation of the workpiece W placed on the belt 31. The processor 11 detects the posture of the workpiece W by principal component analysis on the two-dimensional projected point cloud. For example, the processor 11 calculates the variance-covariance matrix of the two-dimensional projected point cloud, calculates eigenvectors 72 and 73 for the variance-covariance matrix, and detects the posture of the workpiece W based on the eigenvectors 72 and 73.

[0033] In S105 of FIG. 3, the processor 11 executes a frame determination process for determining the object region 42 based on the position and posture of the object detected in S104. For example, the processor 11 detects a rectangular prism frame that contains the three-dimensional point cloud of the upper surface 51 of the workpiece W and the three-dimensional point cloud of the upper part of the side surface 52 for the detection region 41, and determines it as the object region 42. The bottom surface of the object region 42 can be determined by projecting a rectangle 71 that contains the two-dimensional projected point clouds 61 and 62 shown in FIG. 7 and has an orientation corresponding to the posture of the workpiece W, onto the bottom surface of the detection region 41. The orientation of two sides of the rectangle 71 can be determined based on the orientations of the eigenvectors 72 and 73, for example. The height of the object region 42 can be determined based on the point with the highest Z2-axis coordinate among the three-dimensional point cloud of the upper surface 51 of the workpiece W and the three-dimensional point cloud of the upper part of the side surface 52.

[0034] In S106, the processor 11 determines whether or not to continue the process shown in Fig. 3. For example, while the workpiece W is being transported by the belt conveyor 30, the processor 11 determines to continue the process shown in Fig. 3 (S106: YES), and proceeds to the process of S100. The processor 11 acquires a three-dimensional point cloud for the next workpiece W transported by the belt conveyor 30 (S100). Since the detection area 41 has already been set (S101: YES), the processor 11 skips the processes of S102 and S103 and proceeds to the process of S104. On the other hand, in S106, when the transport of the workpiece W by the belt conveyor 30 is completed, the processor 11 determines not to continue the process shown in Fig. 3 (S106: NO), and ends the process of Fig. 3.

[0035] [Modifications] In the above embodiment, the object detection process of S104 has been described in the case where the processor 11 detects the position and orientation of the workpiece W included in the three-dimensional point cloud. However, when the processor 11 performs clustering on the projected point cloud, the number of objects whose positions and orientations are detected is not limited to one. The processor 11 may detect the positions and orientations of multiple objects. When the processor 11 detects the positions and orientations of multiple objects in the object detection process of S104, the processor 11 may determine multiple object regions 42 in the frame determination process of S105.

[0036] In the above embodiment, the shapes of the detection area 41 and the object area 42 are rectangular prisms, but are not limited to rectangular prisms. The shapes of the detection area 41 and the object area 42 may be, for example, polygonal prisms other than rectangular prisms, cylinders, or other cylindrical shapes.

[0037] In the above embodiment, the image capturing unit 20 is a ToF camera, but is not limited to this. For example, the image capturing unit 20 may be a projector type camera having a phase difference detection type auto focus detection function.

[0038] [Provisions] (Clause 1) An object recognition device that recognizes an object, comprising an imaging unit that images a subject within an imaging range, and one or more processors, wherein the one or more processors control the imaging unit to execute a point cloud acquisition process that acquires a three-dimensional point cloud of the subject, a floor surface estimation process that estimates the floor surface on which the object is placed based on the three-dimensional point cloud, and a detection area setting process that sets a cylindrical area whose bottom surface is parallel to the floor surface as a detection area for detecting the object.

[0039] According to the above configuration, by setting the detection area for detecting the object as a cylindrical area with its bottom surface parallel to the floor surface, the three-dimensional point cloud used for detecting the object can be limited, thereby shortening the calculation time for recognizing the object.

[0040] (Clause 2) The detection area is arranged in a first coordinate system in which the center of the column is located at the origin, the first coordinate system has X1-axis and Y1-axis which are parallel to the bottom surface and perpendicular to each other, and a Z1-axis which is perpendicular to the bottom surface, and the detection area setting process is a process of setting the detection area arranged in the first coordinate system to a second coordinate system which is parallel to the floor surface estimated by the floor surface estimation process and has X2-axis and Y2-axis which are perpendicular to each other, and a Z2-axis which is perpendicular to the floor surface, and the one or more processors set a transformation matrix R used in coordinate transformation for rotating the detection area by a rotation angle θ around the Z1-axis in the detection area setting process so that the X1-axis and the Y1-axis are parallel to the X2-axis and the Y2-axis, respectively. θ A unit vector parallel to the Z1 axis and having the origin as its base point is the unit normal vector (a, b, c) of the floor surface. T A transformation matrix R is used for coordinate transformation to rotate the detection area so that it coincides with n and a translation vector U for translating the detection area in order to provide a gap between the bottom surface of the column and the floor surface, the point p is transformed into a point q in a second coordinate system of the detection area by the above equation (1), and the transformation matrix R θ is expressed by the above formula (2), and the transformation matrix R n The object recognition device according to clause 1, wherein is expressed by the above formula (3).

[0041] According to the above configuration, the detection area can be set by the above formula (1) without using external information such as 3D CAD data of the target object.

[0042] (Clause 3) The translation vector U is a vector (t X ,t Y ,t Z ) T and a unit vector n parallel to the X2 axis, the Y2 axis, and the Z2 axis, with the origin of the second coordinate system as the base point. X2 ,n Y2 and n Z2and the object recognition device described in clause 2, expressed by the above equation (4) based on the coordinate C0 of the center of gravity of the floor surface estimated by the floor surface estimation process in the second coordinate system and the height r of the detection area in the Z1-axis direction.

[0043] According to the above configuration, the detection area can be set at a position away from the floor surface without using external information such as 3D CAD data of the target object. By detecting the detection area at a position away from the floor surface, it is possible to prevent the floor surface from being recognized as the target object.

[0044] (Clause 4) The object recognition device described in any one of Clauses 1 to 3, wherein the one or more processors project a point cloud within the detection area set by the detection area setting process, from the three-dimensional point cloud acquired by the point cloud acquisition process, onto the bottom surface of the detection area, perform clustering on the point cloud projected onto the bottom surface of the detection area, and further perform an object detection process to detect the position and posture of the object located within the detection area based on the point clouds before and after projection.

[0045] According to the above configuration, the point cloud within the detection area is projected onto the floor surface, and the three-dimensional point cloud is converted into a two-dimensional point cloud, thereby reducing the time required to detect the position and orientation of the target object.

[0046] (Clause 5) The object recognition device described in Clause 4, wherein the one or more processors, in the object detection process, calculate eigenvectors for the point clouds projected onto the bottom surface of the detection area, detect the orientation of the object based on the eigenvectors, and further perform a frame determination process to determine an object area that contains each point cloud of the object located within the detection area and is parallel to the floor surface based on the position and orientation of the object detected by the object detection process.

[0047] According to the above configuration, by calculating an eigenvector for the point cloud projected onto the bottom surface of the detection area and detecting the posture of the target object based on the eigenvector, it is possible to properly determine the object area even if the target object is placed on an inclined floor surface.

[0048] (Clause 6) An object recognition device described in any one of Clauses 1 to 5, wherein, in the point cloud acquisition process, the one or more processors acquire from the imaging unit an image of the subject and a distance image that visualizes the distance from the imaging unit to the subject, and acquire a three-dimensional point cloud of the subject based on the image and the distance image.

[0049] When the workpiece W placed on the belt 31 is imaged, a shadow may appear on the surface of the workpiece W due to the influence of the external environment. If an attempt is made to acquire a three-dimensional point cloud only from the captured image, the three-dimensional point cloud of the shaded portion may not be properly acquired. In addition, if sand or the like adheres to the surface of the workpiece W and discolors it, the three-dimensional point cloud may not be properly acquired if an attempt is made to acquire a three-dimensional point cloud only from the captured image. According to the above configuration, by acquiring a three-dimensional point cloud of the subject using the captured image and the distance image, it is possible to properly acquire a three-dimensional point cloud even if an abnormality such as a shadow appears in the captured image due to the influence of the external environment.

[0050] (Clause 7) An object recognition method for recognizing an object using an object recognition device described in any one of clauses 1 to 6, comprising: a point cloud acquisition process in which the one or more processors control the imaging unit to acquire a three-dimensional point cloud of the subject; a floor surface estimation process in which the one or more processors estimate the floor surface on which the object is placed based on the three-dimensional point cloud; and a detection area setting process in which the one or more processors set a cylindrical area whose bottom surface is parallel to the floor surface as a detection area for detecting the object.

[0051] According to the above configuration, by setting the detection area for detecting the object to a cylindrical area whose bottom surface is parallel to the floor surface, the three-dimensional point cloud used to detect the object can be limited, thereby shortening the calculation time for recognizing the object.

[0052] (Clause 8) A program for causing a computer to function as an object recognition device described in any one of clauses 1 to 6, the program causing the computer to execute the point cloud acquisition process, the floor surface estimation process, and the detection area setting process.

[0053] According to the above configuration, by setting the detection area for detecting the object to a cylindrical area whose bottom surface is parallel to the floor surface, the three-dimensional point cloud used to detect the object can be limited, thereby shortening the calculation time for recognizing the object. [Software implementation example] The functions of the object recognition device 1 (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (particularly each part included in the control device 10).

[0054] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0055] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0056] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present disclosure. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0057] [Additional Notes] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present disclosure. [Explanation of symbols]

[0058] 1 Object recognition device, 10 Control device, 11 Processor, 12 Primary memory, 13 Secondary memory, 20 Imaging unit, 21 Imaging range, 31 Belt (floor surface), 41 Detection area, 41A Bottom surface, 42 Object area, 61, 62 Projection point cloud, 72, 73 Eigenvector, C2 Center of gravity, n Unit normal vector, O1, O2 Origin, P1 Program, R n , R θ Transformation matrix, U translation vector, W work (object), θ rotation angle

Claims

1. An object recognition device for recognizing an object, an imaging unit that captures an image of a subject within an imaging range; one or more processors; The one or more processors: A point cloud acquisition process of controlling the imaging unit to acquire a three-dimensional point cloud of the subject; a floor surface estimation process for estimating a floor surface on which the object is placed based on the three-dimensional point cloud; A detection area setting process is performed to set a cylindrical area whose bottom surface is parallel to the floor surface as a detection area in which the object is detected.

2. the detection area is disposed in a first coordinate system in which a center of the cylinder is located at an origin, the first coordinate system is parallel to the bottom surface, and the first coordinate system has an X1-axis and a Y1-axis which are perpendicular to each other, and a Z1-axis which is perpendicular to the bottom surface; the detection area setting process is a process of setting the detection area arranged in the first coordinate system into a second coordinate system having an X2-axis and a Y2-axis which are parallel to the floor surface estimated by the floor surface estimation process and perpendicular to each other, and a Z2-axis which is perpendicular to the floor surface; The one or more processors, in the detection area setting process, A transformation matrix R is used for coordinate transformation to rotate the detection area around the Z1 axis by a rotation angle θ so that the X1 axis and the Y1 axis are parallel to the X2 axis and the Y2 axis, respectively. θ and, A unit vector parallel to the Z1 axis and having the origin as its base point is a unit normal vector (a, b, c) of the floor surface. T A transformation matrix R is used in a coordinate transformation to rotate the detection area so that the detection area coincides with the n and, a translation vector U that translates the detection area to provide a gap between the bottom surface of the column and the floor surface; The point p is transformed into a point q in a second coordinate system of the detection area by the following equation (1) using q=R n R θ p + U ... (1) The transformation matrix R θ is expressed by the following formula (2): [0010] …(2) The transformation matrix R n The object recognition device according to claim 1 , wherein: [0025] …(3)

3. The translation vector U is A vector (t X , t Y , t Z ) T and, A unit vector n parallel to the X2 axis, the Y2 axis, and the Z2 axis, with the origin of the second coordinate system as the base point. X2 , n Y2 and n Z2 and, The coordinate C of the center of gravity of the floor surface estimated by the floor surface estimation process in the second coordinate system 0 and, The object recognition device according to claim 2 , which is expressed by the following formula (4) based on a height r of the detection area in the Z1-axis direction. U=C 0 +t X n X2 +t Y n Y2 + (t Z +r / 2)n Z2 …(4)

4. The one or more processors:

2. The object recognition device according to claim 1, further comprising: projecting a point cloud within the detection area set by the detection area setting process, out of the three-dimensional point cloud acquired by the point cloud acquisition process, onto the bottom surface of the detection area; performing clustering on the point cloud projected onto the bottom surface of the detection area; and detecting a position and orientation of the object located within the detection area based on the point clouds before and after the projection.

5. The one or more processors: In the object detection process, an eigenvector is calculated for a group of points projected onto the bottom surface of the detection area, and a posture of the target object is detected based on the eigenvector; The object recognition device according to claim 4, further comprising a frame determination process for determining an object area that includes a point cloud of each of the objects located within the detection area and is parallel to the floor surface, based on the position and orientation of the object detected by the object detection process.

6. The one or more processors:

2. The object recognition device according to claim 1, wherein, in the point cloud acquisition process, an image of the subject and a distance image that visualizes the distance from the imaging unit to the subject are acquired from the imaging unit, and a three-dimensional point cloud of the subject is acquired based on the image and the distance image.

7. 2. An object recognition method for recognizing an object using the object recognition device according to claim 1, comprising: a point cloud acquisition step in which the one or more processors control the imaging unit to acquire a three-dimensional point cloud of the subject; a floor surface estimation step in which the one or more processors estimate a floor surface on which the object is placed based on the three-dimensional point cloud; An object recognition method comprising: a detection area setting step in which the one or more processors set a cylindrical area whose bottom surface is parallel to the floor surface as a detection area in which the object is detected.

8. 2. A program for causing a computer to function as the object recognition device according to claim 1, the program causing the computer to execute the point cloud acquisition process, the floor surface estimation process, and the detection area setting process.

Citation Information

Patent Citations

  • Information processing apparatus, system, detection method, and computer program

    JP2022028196A