Position estimation device

Abstract

To provide a position estimation device which improves accuracy of estimating a position of an object given with a marker.SOLUTION: A position estimation device (20) for estimating a position of an object (O) detects a marker (M) by specifying depth information of the marker (M) and two or more pieces of position information in pieces of position information on four corners of the marker (M) from a detailed image and a depth image of the marker (M) from a first measurement device (10), estimates a position of the detected marker (M) by calculating translation information being vector information and rotation information relative to the first measurement device of the marker (M) from the two or more pieces of position information, calculates a plane distance between the marker (M) and the first measurement device from the translation information, calculates the azimuth of the marker (M) to the first measurement device from the rotation information, and calculates a depth distance being a distance at the depth from the depth information of the marker (M).SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a position estimation device. [Background technology]

[0002] For example, Patent Document 1 discloses a technique for detecting a marker from an image captured by an imaging means and acquiring position information based on the detected marker. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. WO2020 / 013337 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, when a robot is to grasp an object, it is necessary to estimate with high accuracy the position at which the robot can grasp the object based on the position information acquired in this manner. However, in the conventional techniques described above, the accuracy of the acquired position may not be sufficient, for example, as the accuracy of the position used to make the robot grasp the object.

[0005] An object of one aspect of the present invention is to realize a position estimation device that improves the accuracy of the estimated position. [Means for solving the problem]

[0006] A position estimation device for estimating the position of an object, comprising a processor and a first memory, wherein the processor processes, by a program loaded in the first memory, a detailed image and a depth image synchronized in frame relation from a first measurement device comprising a first imager for capturing a detailed image and a first depth measurement device for capturing a depth image, the synchronized detailed image and the depth image capturing a marker attached to the object, and the synchronized detailed image and the depth image include a marker detection step of detecting the marker by identifying position information of two or more of the position information of four corners of the marker and depth information of the marker from the detailed image; Provided is a position estimation device that performs frame-related processing including a marker position estimation step that estimates the position of a detected marker by calculating translation information and rotation information, which are vector information of the marker relative to a first measuring device, from two or more pieces of position information; a plane position calculation step that calculates a plane distance, which is the distance on a plane between the first measuring device and the marker, from the translation information; an orientation calculation step that calculates the orientation of the marker relative to the first measuring device from the rotation information; and a center depth calculation step that calculates a depth distance, which is the distance in depth, from the depth information of the marker. [Effects of the Invention]

[0007] According to one aspect of the present invention, it is possible to realize a position estimation device that improves the accuracy of estimating the position of an object to which a marker is attached. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a hardware block diagram illustrating a configuration of a position estimation system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram illustrating an example of a marker according to the present embodiment. [Figure 3] 10 is a flowchart illustrating an example of a position estimation process. [Figure 4] FIG. 2 is a software block diagram showing each step of the position estimation process as a block. [Figure 5] FIG. 10 is a sequence diagram of a position estimation process. [Figure 6]10 is a flowchart illustrating an example of an averaging process. [Figure 7] FIG. 1 is a schematic diagram illustrating a position estimation system including a plurality of measuring devices. [Figure 8] 10 is a flowchart illustrating an example of a position estimation process when a plurality of measuring devices are provided. [Figure 9] FIG. 2 is a diagram illustrating a display screen of an output device. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment 1) An embodiment of the present invention will be described in detail below. Fig. 1 is a block diagram showing a position estimation system according to this embodiment. The position estimation system includes a measurement device 10, a position estimation device 20, and an output device 30. The measurement device 10 is, for example, a measurement device 10a (hereinafter, this may be referred to as a first measurement device) described below.

[0010] The measurement device 10 has an imager 11, a depth measurement device 12, a processor 13, a memory 14, an input / output interface 15, and a bus 16, and acquires, for example, a detailed image and a depth image of the marker M. The imager 11, the depth measurement device 12, the processor 13, the memory 14, and the input / output interface 15 are interconnected via the bus 16. Unless otherwise noted, there is one marker M.

[0011] The image capturer 11 is, for example, an image sensor, more specifically, a CMOS camera, and captures a detailed image of the marker M based on reflected light L0 from the marker M. The depth measurement device 12 is, for example, a depth sensor, more specifically, a ToF (Time of Flight), and measures a depth image of the marker M. The measurement device 10 is, for example, a REALSENSEL515 LiDAR from Intel with a resolution of w1280h720.

[0012] The detailed image mainly represents planar information, which is information about the plane of the image, but also includes depth information indicating the depth in the image as brightness, etc., and the depth image mainly represents depth information but also includes planar information.

[0013] Specifically, the depth information represents the distribution of distances (three-dimensional distances) between the marker M and the measurement device 10 (depth measurement device 12), and corresponds to the three-dimensional shape of the marker M as seen from the depth measurement device 12.

[0014] The depth measurement device 12, for example, irradiates an electromagnetic wave Lr (beacon, for example, laser light) and measures the distribution of the distance between the measurement device 10 (depth measurement device 12) and the marker M based on the electromagnetic wave Lr reflected by the marker M. In other words, the measurement device 10 (depth measurement device 12) irradiates the marker M with the electromagnetic wave Lr and derives depth information based on the electromagnetic wave Lr reflected from the marker M.

[0015] A program P1 is stored as, for example, non-volatile storage in the memory 14. The processor 13 executes processing related to acquisition of a planar image and depth information of the marker M in accordance with instructions included in the program P1.

[0016] The processor 13 acquires detailed images and depth images from the imager 11 and depth meter 12, synchronizes the detailed images and depth images, and outputs them to the position estimation device 20 via the input / output interface 15, for example, frame by frame.

[0017] For example, the measurement device 10 processes each frame at a predetermined time interval such as 1 / 60 seconds or 1 / 30 seconds. Note that the detailed image and depth image between different frames may be synchronized.

[0018] Here, the plane information and depth information can be expressed in coordinates at the time of measurement. The coordinates at the time of measurement are XYZ coordinates based on the measurement device 10. The Z axis is an axis in the direction from the measurement device 10 toward the marker M, and the XY plane is a plane perpendicular to the Z axis.

[0019] Here, the processor 13 may adjust the angle of view and complement the depth information as follows: Adjusting the angle of view means aligning the angles of view of the image capture device 11 and the depth measurement device 12 so that they match.

[0020] That is, when the range (angle of view) or resolution of the detailed image and the depth image differ, the range or resolution is adjusted. This allows the detailed image and the depth image to correspond to each other. Specifically, the correspondence between the planar information and the depth information at the location of the planar information can be clarified.

[0021] Complementing depth information means complementing depth information when part of the depth information is missing due to poor reflection in part of the marker M. That is, the depth information is complemented by complementing the missing part from the depth information around the missing part or by smoothing the depth information around this part.

[0022] The processor 13 is, for example, a CPU (Central Processing Unit), and the memory 14 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory).

[0023] The input / output interface 15 is, for example, a PCI (Peripheral Component Interconnect) interface. The input / output interface 15 is connected to a position estimation device 20.

[0024] The position estimation device 20 is a device that estimates the position of an object O, and is realized using, for example, a general-purpose computer, and as shown in Fig. 1, includes a processor 21, a first memory (which may be called a primary memory) 22, a second memory (which may be called a secondary memory) 23, an input / output interface 24, and a bus 25. The processor 21, the first memory 22, the second memory 23, and the input / output interface 24 are connected to each other via the bus 25.

[0025] The second memory 23 stores the program P2 and the table T as, for example, non-volatile storage. The processor 21 loads the program P2 stored in the second memory 23 onto the first memory 22.

[0026] The processor 21 executes, as a position estimation process, frame-related processing S10 (described later), which is processing for each frame performed in response to processing for each frame of the measuring device 10, in accordance with instructions included in the program P1 deployed on the first memory 22. In a broad sense, the frame-related processing S10 may also include processing related to frames of the measuring device 10.

[0027] Table T stores information used for position estimation, such as the size and the display pattern to be displayed in the center (hereinafter, this may be simply referred to as the display pattern). Table T may also store the ID of the marker M (marker ID).

[0028] For example, the marker ID is associated one-to-one with a combination of information such as the name of the object O and annotations about the object O (for example, the expiration date of the object O), which will be described later, and is saved as a table T.

[0029] The input / output interface 24 is connected to the measurement device 10 and the output device 30. Note that the table T may hold a display pattern including black squares that form frames.

[0030] The processor 21 is, for example, a CPU. The first memory 22 is, for example, a volatile memory, such as a RAM (Random Access Memory). The second memory 23 is, for example, a HDD (Hard Disk Drive) or an SSD (Solid State Drive). The input / output interface 24 is, for example, a PCI interface.

[0031] The output device 30 is, for example, a robot arm A, an image display device B, or an audio output device, which will be described later, and is controlled by the position estimation device 20. Note that, for example, the position estimation device 20 may include the measurement device 10, the processor 21 may include the processor 13, and the first memory 22 and the second memory 23 may include the memory 14.

[0032] 2 is a diagram showing an example of a marker M according to this embodiment. The marker M satisfies, for example, a predetermined condition, and is, for example, an AR marker (for example, an ArUco marker), and is used by being attached to an object O. The marker M is represented by a binary value, and for example, a plurality of white or black squares are arranged two-dimensionally, vertically and horizontally.

[0033] Here, the 7x7 square display pattern is the entity of the marker M and functions as dictionary data for identifying the marker M. Here, black squares are placed around the 7x7 squares as a frame, and the marker M appears to be composed of 9x9 squares.

[0034] The display pattern of the marker M has rotational asymmetry and can be used to calculate the orientation (attitude, for example, roll angle φ, pitch angle θ, yaw angle ψ) of the marker M. The marker M has a Hamming distance that indicates its error resistance, and it is desirable that the minimum Hamming distance, which is the smallest Hamming distance, is as large as possible.

[0035] The display pattern is not limited to a 7x7 grid, and may be, for example, a 5x5 to 7x7 grid. The marker M is preferably a square with a side length of 3 to 100 mm and is expressed in two values, but the shape does not have to be square, and the aspect ratio may be 1:2 or 2:1, for example. In this way, the marker M is assumed to satisfy predetermined conditions in size and display pattern.

[0036] Furthermore, for example, it is preferable that the distance between the marker M and the measuring device 10 is 20 cm to 9 m, and it is preferable that the number of identifiable markers M is up to 1000. Therefore, the position estimation device 20 can simultaneously estimate the positions of up to 1000 markers M, and can recognize 1000 objects O.

[0037] Whether the marker M has been clearly captured can be determined by correcting distortion and adjusting the size of the marker M captured in the detailed image, for example, using a projective transformation, and then calculating the variance of the results processed with a Laplacian filter, and checking whether the result satisfies certain conditions.

[0038] 3 is a flowchart showing an example of a position estimation process (frame-related process S10) performed by the position estimation device 20. The frame-related process S10 is a process executed by, for example, the processor 21, and includes a marker detection step S11, a marker position estimation step S12, a planar position calculation step S13, an orientation calculation step S14, and a center depth calculation step S16. The frame-related process S10 may further include a center coordinate calculation step S15.

[0039] The position estimation device 20 processes, for example, detailed images and depth images synchronized for each frame from a first measurement device 10a including a first imager 11 (which may be simply referred to as the imager 11) that captures detailed images and a first depth measurement device 12 (which may be simply referred to as the depth measurement device 12) that captures depth images. Note that, for example, a marker M attached to an object O is captured in the synchronized detailed images and depth images.

[0040] The processor 21 detects the marker M by identifying two or more pieces of position information (X1, Y1) to (X4, Y4) of the four corners of the marker M, which are planar information, and depth information of the marker M from the detailed image (marker detection step S11).

[0041] Note that the two or more pieces of position information (X1, Y1) to (X4, Y4) of the four corners of the marker M may be all four corners, or may be two pieces of position information on a diagonal line, etc. In addition, in the marker detection step S11, the marker ID may also be identified.

[0042] The processor 21 estimates the position of the detected marker M by calculating translation information and rotation information, which are vector information of the marker M relative to the first measuring device 10a, from the position information of two or more of the four corners of the marker M, (X1, Y1) to (X4, Y4) (marker position estimation step S12).

[0043] The processor 21 calculates a planar distance, which is the distance on a plane between the first measuring device 10a and the marker M, from the translation information (planar position calculation step S13). The processor 21 calculates an orientation of the marker M relative to the first measuring device from the rotation information (orientation calculation step S14).

[0044] The processor 21 calculates a depth distance, which is a distance in depth, from the depth information of the marker M (center depth calculation step S16), and ends the frame-related processing S10. For example, in the center depth calculation step S16, the processor 21 calculates the depth distance by averaging the depth information of two or more of the four corners.

[0045] As another example, in the central depth calculation step S16, the processor 21 calculates the depth distance by offsetting one piece of depth information toward the center using the depth information and orientation of one of the four corners.

[0046] In the central depth calculation step S16, the processor 21 may calculate a depth distance, which is a distance in the depth direction, from the central coordinates (Xc, Yc) and the depth information of the marker M.

[0047] For example, the processor 21 may calculate the center coordinates (Xc, Yc) of the marker M on the plane, which is plane information, from two or more pieces of position information (X1, Y1) to (X4, Y4) of the four corners of the marker M (center coordinate calculation step S15).

[0048] After the frame-related processing S10, the processor 21 performs output processing to output, for example, output information for controlling the output device 30 to the output device 30. The output information is composed of, for example, plane coordinates calculated from the plane distance and depth coordinates calculated from the depth distance.

[0049] In the output process, in addition to the output information, detailed output information including a marker ID, a direction, and the like may be output to the output device 30. The output information is three-dimensional information including planar information and depth information.

[0050] For example, when using position information of all four corners, the center coordinates (Xc, Yc) can be calculated from the position information (X1, Y1) to (X4, Y4) of the four corners of the marker M using the following equations (1) and (2). Xc=(X1+…+X4) / 4 …Formula (1) Yc=(Y1+...+Y4) / 4...Formula (2)

[0051] Note that the planar position calculation step S13 and the orientation calculation step S14 may be executed in the reverse order from that shown in Fig. 3 as long as they are executed after the marker position estimation step S12. Also, the central depth calculation step S16 may be executed, for example, after the planar position calculation step S13 as long as they are executed after the center coordinate calculation step S15. Also, the marker position estimation step S12 and the center coordinate calculation step S15 may be executed in the reverse order from that shown in Fig. 3 as long as they are executed after the marker detection step S11.

[0052] 4 is a software block diagram showing each step of the position estimation process as a block. Specifically, for example, the marker detection step S11 is performed by the marker detection unit 211, the marker position estimation step S12 is performed by the marker position estimation unit 212, and the planar position calculation step S13 is performed by the planar position calculation unit 213.

[0053] Also, for example, the orientation calculation step S14 is performed by the orientation calculation unit 216, the center coordinate calculation step S15 is performed by the center coordinate calculation unit 214, the center depth calculation step S16 is performed by the center depth calculation unit 215, and the output processing is performed by the output processing unit 217.

[0054] For example, in the measurement device 10, the detailed image acquisition unit 131 executes a detailed image acquisition step of acquiring a detailed image from the image capture device 11, and the depth image acquisition unit 132 executes a depth image acquisition step of acquiring a depth image from the depth measurement device 12. Furthermore, the synchronization processing unit 133 executes a synchronization step of synchronizing the detailed image and the depth image synchronized for each frame, for example.

[0055] For example, each of the units such as the detailed image acquisition unit 131, the depth image acquisition unit 132, the synchronization processing unit 133, the marker detection unit 211, the marker position estimation unit 212, the plane position calculation unit 213, the center coordinate calculation unit 214, the center depth calculation unit 215, the orientation calculation unit 216, and the output processing unit 217 may be a module (function) stored in the programs P1 and P2, or may be implemented by a logic circuit such as an FPGA (Field Programmable Gate Array). Note that only a part of each unit may be implemented by an FPGA or the like.

[0056] Figure 5 is a sequence diagram of the position estimation process, and shows an example of the flow of information such as depth image, detailed image, position information of two or more of the four corners of marker M (X1, Y1) to (X4, Y4), depth information, translation information, rotation information, planar distance, depth distance, output information, marker ID, and orientation at each step in the position estimation process.

[0057] As described above, the position estimation device 20 combines a planar distance based on a detailed image acquired from the image capture device 11 with a depth distance based on a depth image acquired from the depth measurement device 12. The position estimation device 20 also estimates the position of the object O by recognizing a marker M that is attached to the object O and satisfies a predetermined condition. In this way, the position estimation device 20 in this embodiment can estimate the position of the object O with high accuracy.

[0058] The predetermined conditions that the marker M must satisfy are, for example, that the size is 3 to 100 mm on a side, that the display pattern is a 5×5 to 7×7 grid, and that the marker M has rotational asymmetry.

[0059] 6 is a flowchart showing an example of the averaging process. The processor 21 may execute an average value calculation process S20 in which, if the output information (detailed output information) output as a result of the frame-related process S10 satisfies a predetermined condition, the output information is regarded as normal output information (normal detailed output information), and an average of a predetermined number or more of the normal output information (normal detailed output information) is output as average normal output information (average normal detailed output information).

[0060] The predetermined condition that the output information satisfies refers to, for example, that the standard deviation of the output information falls within a predetermined range. For example, the average value calculation process S20 (other than the frame-related process S10) is executed by the output processing unit 217.

[0061] Specifically, in the average value calculation process S20, the processor 21 executes the frame-related process S10. The processor 21 determines whether or not the output information output as a result of the frame-related process S10 or the like satisfies a predetermined condition and is not an abnormal value (S21).

[0062] If the output information is determined to be an abnormal value (S21: NO), the processor 21 executes the frame-related process S10. If the output information is determined to be a normal value, not an abnormal value, but satisfying a predetermined condition (S21: YES), the processor 21 stores the normal value as normal output information in, for example, the first memory 22 (S22).

[0063] The processor 21 determines whether or not a predetermined number or more of normal output information is stored in the first memory 22 (S23). If it is determined that the predetermined number or more of normal output information is stored (S23: YES), the processor 21 outputs, for example, average normal output information which is the average of the normal output information, and ends the average value calculation process S20. The processor 21 handles the normal output information stored in the first memory 22, for example, in a FIFO (First In, First Out) format.

[0064] When it is determined that the predetermined number or more are not stored (S23: NO), the processor 21 executes the frame-related process S10 again. As described above, when executing the average value calculation process S20, the position estimation device 20 can exclude abnormal values, thereby further improving the accuracy of position estimation.

[0065] 7 is a schematic diagram showing a position estimation system including a plurality of measuring devices 10. A case will be described in which a position estimation device 20 is connected to a plurality of measuring devices 10, such as a first measuring device 10a and a second measuring device 10b, and a plurality of markers M (marker Ma, marker Mb) are arranged.

[0066] The measuring device 10a can detect the marker Ma but cannot detect the marker Mb because, as viewed from the measuring device 10a, the marker Mb is located in the shadow of the robot arm A. On the other hand, the measuring device 10b can detect both the markers Ma and Mb.

[0067] For example, the robot arm A, which includes the arm body 31 and the suction hand 32, is a device that moves the arm body 31 based on output information from the position estimation device 20 and sucks the target object O with the suction hand 32.

[0068] This also applies to cases where a structure such as a pillar of a building that does not exchange information with the position estimation device 20 is the cause of the inability to detect the marker Mb, not limited to an output device 30 such as the robot arm A.

[0069] In order to prevent the electromagnetic waves Lr from the multiple measurement devices 10 (depth measurement devices 12) from interfering with each other's measurements, it is preferable that the position estimation device 20 executes a multiple measurement device compatible process S30 as shown in Fig. 8. For example, the multiple measurement device compatible process S30 (other than the frame-related process S10) is executed by the output processing unit 217.

[0070] If the multiple measurement device compatible processing S30 is not executed and the normal frame-related processing S10 is executed in parallel, for example, the first measurement device 10a (depth measurement device 12) may unintentionally perform measurements using the electromagnetic waves Lr from the second measurement device 10b, resulting in unintentional continuous measurements.

[0071] In the multiple measurement device handling process S30, in response to an instruction from the position estimation device 20, the measurement device 10 (depth measurement device 12) switches ON / OFF of the irradiation of the electromagnetic wave Lr like a shutter.

[0072] In the multiple measurement device compatible processing S30, the processing time for one frame is variable rather than fixed, unlike when using only one measurement device 10. Specifically, in the multiple measurement device compatible processing S30, the position estimation device 20 performs processing such that when processing for one frame for one measurement device 10 is completed, processing for one frame for another measurement device 10 is started.

[0073] 8 is a flowchart showing an example of a multiple measurement device support process S30, which is a position estimation process when multiple measurement devices are provided. In the multiple measurement device support process S30, the processor 21 issues a synchronization setting instruction to the first measurement device 10a and the second measurement device 10b, and executes the frame-related process S10 for each of the first measurement device 10a and the second measurement device 10b in turn.

[0074] In the multiple measurement device support process S30, the processor 21 may execute the average value calculation process S20 instead of the frame-related process S10. Also, for example, the multiple measurement device support process S30 is started when multiple measurement devices 10 are connected to the position estimation device 20.

[0075] Specifically, in the multiple measurement device handling process S30, the processor 21 transmits a synchronization setting instruction to the measurement device 10 (S31). The processor 21 determines whether or not the synchronization setting instruction has been transmitted to all measurement devices 10 registered in the position estimation device 20 (S32).

[0076] If it is determined that the synchronization setting instruction has not been transmitted to all of the measuring devices 10 registered in the position estimation device 20 (S32: NO), the processor 21 executes step S31. Note that registration information indicating whether or not a measuring device 10 is registered in the position estimation device 20 is assumed to be stored in, for example, table T of the second memory 23.

[0077] If it is determined that the synchronization setting instruction has been transmitted to all of the measurement devices 10 registered in the position estimation device 20 (S32: YES), the processor 21 transmits a control start instruction to the measurement devices 10 (S33). The processor 21 executes frame-related processing S10 for the measurement devices 10 that transmitted the control start instruction.

[0078] The processor 21 determines whether or not the control start instruction has been transmitted to all of the measuring devices 10 registered in the position estimation device 20 (S34). If it is determined that the control start instruction has not been transmitted to all of the measuring devices 10 registered in the position estimation device 20 (S34: NO), the processor 21 executes step S33.

[0079] If it is determined that a control start instruction has been sent to all measuring devices 10 registered in the position estimation device 20 (S34: YES), the processor 21 outputs, for example, output information regarding each measuring device 10, and terminates the multiple measuring device support processing S30.

[0080] As described above, when the position estimation device 20 executes the multiple measurement device compatible processing S30, it is possible to select information acquired from each of the measurement devices 10a and 10b, for example, by rejecting abnormal values, acquiring information from the measurement device 10a that is closer to the marker M, or taking an average.

[0081] Therefore, the position estimation device 20 can reduce blind spots, which are areas where the position cannot be estimated, and can more accurately estimate the position of the object O. Note that the above-mentioned adjustment and interpolation may be performed across the first measuring device 10a and the second measuring device 10b.

[0082] Furthermore, in the multiple measurement device processing S30, detailed images and depth images are captured by each measurement device 10 in accordance with the ON / OFF of the irradiation of the electromagnetic wave Lr. By varying the timing of the irradiation of the electromagnetic wave Lr from the multiple measurement devices 10 (depth measurement devices 12), it becomes possible to perform measurements without unintended measurements by the multiple measurement devices 10 (depth measurement devices 12). For example, unintended measurement refers to measurement in a state where interference (a state in which ripple-like noise is superimposed on the depth image, preventing normal measurement in the first place) occurs.

[0083] 9 is a diagram showing a display screen of an image display device B, such as a display, which is the output device 30. Output information and orientation, which are composed of plane coordinates calculated from the plane distances and depth coordinates calculated from the depth distances output by the frame-related processing S10 or the like, are displayed near the object O on the display screen. A marker ID may also be displayed near the object O on the display screen.

[0084] Note that the name of the object O and notes about the object O (for example, the expiration date of the object O) may be displayed as detailed output information. The object O is, for example, a sandwich placed as a commodity on a sales shelf in a retail store such as a convenience store. A plurality of markers M (markers Ma, Mb, Mc) are attached to the object O.

[0085] On the display screen, a detailed image captured by at least one of the first measurement device 10a and the second measurement device 10b and output information and detailed output information for the marker M are displayed in a superimposed manner.

[0086] The display screen may also display images of the robot arm A and the measuring device 10. The display screen may also reflect information transmitted from the position estimation device 20 over time.

[0087] Marker information is displayed beside each of the markers M (Ma, Mb, Mc) attached to the sandwich (object O). For example, the markers M are attached to each imaged surface of the sandwich package. Output information and detailed output information for the markers M are displayed superimposed near the markers M (object O). Note that the markers M may be displayed with their squares or outlines emphasized.

[0088] For example, on the display screen, the output information is displayed as millimeters (mm) for each of the X, Y, and Z coordinates, the name of the object O is displayed in English letters, and the expiration date of the object O is displayed in English letters and numbers, etc. For example, on the display screen, the marker ID is displayed in numbers, and the orientation is displayed as the roll angle φ, pitch angle θ, and yaw angle ψ in degrees (deg).

[0089] As described above, when the display screen is displayed in the position estimation device 20 in this embodiment, the user of the position estimation device 20 can visually see what information is being acquired, making it easier for the user to adjust or change the position estimation device 20 or the measuring device 10.

[0090] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

[0091] For example, the marker M is not limited to an AR marker, but may be a two-dimensional code, a one-dimensional code such as a barcode, or something like a photograph, and may be represented by three or more colors instead of a binary value.

[0092] Furthermore, if the object O satisfies certain conditions, such as having a uniform shape that allows easy identification of multiple patterns, and having rotational asymmetry that allows calculation of the orientation, the marker M may be the object O itself. Note that the number of measuring devices 10 may be three or more.

[0093] Note that an audio output device such as a speaker may be connected as the output device 30 to the position estimation device 20. For example, when the robot arm A approaches the target marker M, the output device 30 may output a warning sound as a warning that the robot arm A will adsorb the target object O.

[0094] The pitch or volume of the warning sound output from the output device 30 may change depending on, for example, the distance between the robot arm A and the target O (marker M). Specifically, the warning sound output from the output device 30 may increase in volume or pitch as the distance between the robot arm A and the target O decreases.

[0095] When processing a plurality of markers Ma, Mb, and Mc, the position estimation device 20 processes each of the plurality of markers Ma, Mb, and Mc one by one in order.

[0096] When multiple markers Ma, Mb, and Mc are attached to one object O, the position estimation device 20 can recognize more than 1000 objects O. For example, by using marker IDs 001 to 099 for position identification and marker IDs 0100 to 0200 for expiration date definition, the position estimation device 20 can recognize more than 1000 objects O.

[0097] Also, consider a case where marker ID 004 indicates vertex 1 of an onigiri, which is an object O having three vertices, marker ID 005 indicates vertex 2 of the onigiri, and marker ID 006 indicates vertex 3 of the onigiri.

[0098] In this case, the position estimation device 20 can recognize that the center of the position information of the multiple marker IDs is the position of the rice ball. For example, the marker ID of marker Ma is 004, the marker ID of marker Mb is 005, and the marker ID of marker Mc is 006. [Explanation of symbols]

[0099] 10...measuring device, 11...imager, 12...depth measuring device, 20...position estimation device, 30...output device, M...marker, O...object.

Claims

1. A position estimation device for estimating a position of an object, comprising a processor and a first memory, the processor processes the synchronized detailed images and the synchronized depth images from a first measurement device including a first imager for capturing detailed images and a first depth measurement device for capturing depth images, using a program loaded in the first memory; A marker that satisfies a predetermined condition attached to the object is captured in the synchronized detailed image and the synchronized depth image, a marker detection step of detecting the marker by identifying position information of two or more of four corners of the marker and depth information of the marker from the detailed image; a marker position estimation step of estimating a position of the detected marker by calculating translation information and rotation information, which are vector information of the marker with respect to the first measuring device, from position information of two or more of the four corners of the marker; a plane position calculation step of calculating a plane distance, which is a distance on a plane between the first measuring device and the marker, from the translation information; an orientation calculation step of calculating an orientation of the marker relative to the first measuring device from the rotation information; a center depth calculation step of calculating a depth distance, which is a distance in depth, from the depth information of the marker; Run The processor: When the output information output by the frame-related processing, which is composed of the plane coordinates calculated from the plane distances and the depth coordinates calculated from the depth distances, satisfies a predetermined condition, the output information is determined to be normal; A position estimation device that outputs an average of a predetermined number or more of normal output information as average normal output information.

2. A position estimation device comprising a processor and a first memory, which estimates the position of an object, the processor processes the synchronized detailed images and the synchronized depth images from a first measurement device including a first imager for capturing detailed images and a first depth measurement device for capturing depth images, using a program loaded in the first memory; A marker that satisfies a predetermined condition attached to the object is captured in the synchronized detailed image and the synchronized depth image, a marker detection step of detecting the marker by identifying position information of two or more of four corners of the marker and depth information of the marker from the detailed image; a marker position estimation step of estimating a position of the detected marker by calculating translation information and rotation information, which are vector information of the marker with respect to the first measuring device, from position information of two or more of the four corners of the marker; a plane position calculation step of calculating a plane distance, which is a distance on a plane between the first measuring device and the marker, from the translation information; an orientation calculation step of calculating an orientation of the marker relative to the first measuring device from the rotation information; a center depth calculation step of calculating a depth distance, which is a distance in depth, from the depth information of the marker. Run A position estimation device in which output coordinates composed of plane coordinates calculated from the plane distances and depth coordinates calculated from the depth distances output by the frame-related processing are displayed near the object on a display screen.

3. The processor: issuing a synchronization setting instruction to the first measurement device and the second measurement device, and executing the frame-related processing for the first measurement device and the second measurement device, respectively, in sequence; The position estimation device according to claim 1 or 2.

4. The markers that satisfy the predetermined conditions have a display pattern of 5x5 to 7x7 squares and are expressed in two values. The position estimation device according to claim 1 or 2.

Images