Mobile target identification system, mobile target identification device, target identification method and program
The system efficiently estimates pallet position and orientation by recognizing front surface corners and estimating rear surface corners using machine learning and PnP problem solving, addressing inaccuracies in existing methods and enhancing forklift operation and autonomy.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- NEC CORP
- Filing Date
- 2022-11-17
- Publication Date
- 2026-07-29
AI Technical Summary
Existing technologies struggle to accurately estimate the position and orientation of pallets being transported by moving objects, such as forklifts, due to limitations in symmetry-based and line segment-based estimation methods.
A system utilizing a holding means, height acquisition, recognition, search, and estimation means to identify the position and orientation of pallets by recognizing corners on the front surface and estimating corresponding corners on the rear surface using machine learning and PnP problem solving.
Enables accurate and efficient estimation of pallet position and orientation, even when the rear surface is not visible, contributing to improved operator assistance and autonomous driving of forklifts.
Smart Images
Figure 0007896693000003 
Figure 0007896693000004 
Figure 0007896693000005
Abstract
Description
[Technical Field]
[0001] This disclosure relates to a mobile target identification system, a mobile target identification device, a target identification method, and program Regarding. [Background technology]
[0002] In systems where a moving object transports pallets, there are techniques for determining the position and orientation of the pallet. Patent Document 1 describes estimating the position and orientation of a pallet based on line segments of a bounding box (hereinafter referred to as BB) surrounding either the front surface of the pallet or one of two holes, as captured by a camera. It also describes that the position and orientation of the pallet may be estimated based on BB data and reference data.
[0003] Patent Document 2 describes determining whether a forklift is facing a pallet directly based on whether the shape of the pallet included in the image is symmetrical. [Prior art documents] [Patent Documents]
[0004] [Patent Document 1] Japanese Patent Publication No. 2021-24718 [Patent Document 2] Japanese Patent Publication No. 2020-109030 [Overview of the project] [Problems that the invention aims to solve]
[0005] In the invention described in Patent Document 1, the position and orientation of the pallet are calculated by comparing the length of the line segment of BB and the BB data with reference data, so there is a possibility that the position and orientation of the pallet cannot be estimated with high accuracy.
[0006] The invention described in Patent Document 2 is based on whether or not the shape of the pallet included in the image is symmetrical, and therefore it cannot estimate the position and orientation of a pallet that is not directly facing the forklift.
[0007] Therefore, the inventions described in Patent Documents 1 and 2 may not be able to efficiently estimate the position and orientation of the pallet. [Means for solving the problem]
[0008] The system for identifying objects to be moved in this disclosure is: A holding means for holding the object to be moved, A height acquisition means for acquiring the height of the imaging device attached to the holding means, A recognition means for recognizing the position of a corner on the front surface of the moving object captured using the aforementioned imaging device, A search means for searching for similar data to the image of the moving object, depending on the height of the imaging device acquired and the position of the corner present on the front of the recognized moving object, Estimation means for estimating the position of the corner present on the rear surface of the moving object according to the similar data that has been searched, A means for identifying the state of the moving object according to the position of the corners on the front of the recognized moving object and the position of the corners on the rear of the moving object estimated, A system for identifying moving targets equipped with That is the case.
[0009] The mobile target identification device in this disclosure is A holding means for holding the object to be moved, A height acquisition means for acquiring the height of the imaging device attached to the holding means, A recognition means for recognizing the position of a corner on the front surface of the moving object captured using the aforementioned imaging device, A search means for searching for similar data to the image of the moving object, depending on the height of the imaging device acquired and the position of the corner present on the front of the recognized moving object, Estimation means for estimating the position of a corner existing on the rear surface of the object to be moved according to the similar data explored; Specification means for specifying the state of the object to be moved according to the position of a corner existing on the front surface of the recognized object to be moved and the position of a corner existing on the rear surface of the estimated object to be moved; It is an object specifying device provided with these.
[0010] The object specifying method of the present disclosure captures an image of an object using an imaging device; acquires the height of the imaging device; recognizes the position of a corner existing on the front surface of the object imaged using the imaging device; searches for similar data similar to the image of the object according to the acquired height of the imaging device and the position of the corner existing on the front surface of the recognized object; estimates the position of a corner existing on the rear surface of the object according to the explored similar data; It is an object specifying method for specifying the state of the object according to the position of a corner existing on the front surface of the recognized object and the position of a corner existing on the rear surface of the estimated object.
[0011] [[ID=This disclosure provides a moving object identification system that can efficiently estimate the position and orientation of a pallet. [Brief explanation of the drawing]
[0013] [Figure 1] This is a schematic diagram of a moving target identification system according to an embodiment. [Figure 2] This is a block diagram showing the configuration of the moving target identification system according to the embodiment. [Figure 3] This is a flowchart of the target identification method according to the embodiment. [Figure 4] This is a block diagram of the moving target identification system according to Embodiment 1. [Figure 5] This diagram shows the hierarchy of the dictionary data section according to Embodiment 1. [Figure 6] This is a flowchart of the target identification method according to Embodiment 1. [Figure 7] This figure shows the recognition of the front corner position, the search for dictionary data, and the estimation of the rear corner position according to Embodiment 1. [Figure 8] This figure shows how to determine the position and orientation of a moving object from the positions of the front corners and rear corners according to Embodiment 1 by solving a PnP problem. [Modes for carrying out the invention]
[0014] Embodiment Embodiments of the present invention will be described below with reference to the drawings. However, the invention claimed is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential for solving the problem. For clarity of explanation, the following descriptions and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.
[0015] (Description of the moving target identification system in the embodiment) Figure 1 is a schematic diagram of the moving object identification system according to an embodiment. Figure 2 is a block diagram showing the configuration of the moving object identification system according to an embodiment. The moving object identification system according to an embodiment will be described with reference to Figures 1 and 2.
[0016] As shown in Figure 1, the moving object identification system 100 according to this embodiment comprises a moving body 101, a holding unit 102, an imaging device 103, a sensor 104, and an information processing device 105.
[0017] The mobile body 101 is, for example, a forklift. The mobile body 101 can transport and move a movable object 701 (see Figure 7) of a fixed shape using the holding part 102. The mobile body 101 itself does not need to move. The mobile body 101 only needs to be able to change the height of the movable object 701 using the holding part 102. The movable object 701 is, for example, a pallet and has a fixed size. The movable object 701 is a loading platform for transporting goods. The front of the movable object 701 is a rectangular parallelepiped. That is, there are four corner positions on the front of the movable object 701. Similarly, there are four corner positions on the rear of the movable object 701. The movable object 701 has insertion holes at the corners of its front, and can be lifted by inserting the holding part 102 into these holes.
[0018] The holding part 102 is, for example, a fork attached to a forklift. The holding part 102 has an L-shape when viewed from the side, and its bottom is inserted into the object to be moved 701, thereby holding the object to be moved 701. The holding part 102 can be moved up and down. Therefore, the height of the holding part 102 can be changed.
[0019] The imaging device 103 is, for example, an RGB-D camera. An RGB-D camera is a camera that outputs depth data and color data. Alternatively, for example, an RGB camera and a depth sensor may be used as the imaging device 103. There may be multiple imaging devices 103. The imaging device 103 is attached to the holding unit 102 and images the area around the imaging device 103 and the moving object 701. The imaging device 103 may be configured to enable driving assistance or automatic driving by imaging the area around the imaging device 103. In addition, the imaging device 103 can determine the position and orientation of the moving object 701 by imaging the moving object 701, as will be described later.
[0020] Sensor 104 is a variety of sensors that sense the state of the moving body 101 or the holding unit 102. Since the holding unit 102 is attached to the moving body 101, sensor 104 specifically acquires the height of the holding unit 102 relative to the moving body 101. Sensor 104 may also acquire the height of the holding unit 102 based on the operation information of the lift cylinder of the moving body 101. Alternatively, sensor 104 may measure the height of the holding unit 102 from the ground. Sensor 104 may also measure the height from the ground by attaching a distance-measuring sensor such as a LiDAR, laser sensor, radar sensor, or ToF sensor to the holding unit 102. Since the imaging device 103 is attached to the holding unit 102, acquiring the height of the holding unit 102 is equivalent to acquiring the height of the imaging device 103. By acquiring the height of the holding unit 102 when the moving object 701 is imaged using the imaging device 103, the height of the imaging device 103 at the time of image acquisition can be obtained. The sensor 104 may also sense the position, speed, and distance from the moving object 701 to the holding unit 102 of the moving body 101 for the purpose of assisting in the operation of the moving body 101 or for automatic operation.
[0021] The information processing device 105 processes data collected from various devices and sensors attached to the mobile body 101. The information processing device 105 is connected to the mobile body 101 via a network, such as Wi-Fi® or Bluetooth®. The information processing device comprises at least one processor for executing instructions and at least one memory for storing instructions. For example, the information processing device 105 acquires images from the imaging device 103. The information processing device 105 also acquires sensor information from the sensor 104. Furthermore, the information processing device 105 issues control commands to the mobile body 101 to control the mobile body 101 for driving assistance or automatic driving. The information processing device 105 may include, for example, a machine learning machine. The information processing device 105 may also distribute some or all of its functions to the cloud. Furthermore, the information processing device 105 may consist of one device or multiple devices. Here, we have shown that the information processing device 105 remotely controls the mobile body 101, but the information processing device 105 may also be installed on the mobile body 101 and the mobile body 101 may operate independently. In this case, it can be considered as a single mobile target identification device.
[0022] If the position and orientation of the moving object 701 can be determined, it will be possible to assist the operator of the mobile unit 101. Furthermore, it will contribute to the realization of autonomous driving of the mobile unit 101.
[0023] The processing of the information processing device 105 according to the embodiment will be explained with reference to Figure 2. As shown in Figure 2, the information processing device 105 includes an image acquisition unit 201, a height acquisition unit 202, a recognition unit 203, a search unit 204, an estimation unit 205, and a target identification unit 206.
[0024] The image acquisition unit 201 acquires an image from the imaging device 103 attached to the holding unit 102. The image from the imaging device 103 may be a normal RGB image without depth information. The image also includes the moving object 701.
[0025] The height acquisition unit 202 acquires the height of the holding unit 102 when imaging is performed. The height of the holding unit 102 when imaging is performed is the same as the height of the imaging device 103 when imaging is performed.
[0026] The recognition unit 203 recognizes the positions of the corners on the front of the moving object 701 from the captured image. As mentioned above, it recognizes the positions of the four corners on the front of the moving object 701, which has a rectangular parallelepiped shape. Here, "recognizing" the positions of the four corners means identifying the locations where the four corners exist. Known methods can be used for recognition. For example, the front of the pallet can be cut out using the recognition result of the pallet holes, and the position P of the corners on the front of the pallet can be determined using edge detection. k It may be recognized. Edge detection is a method of recognizing discontinuous changes in an image based on the amount of change in features within the image. In addition, feature point information of the corner positions is pre-machine learned, and using feature point matching, the image of the moving target 701 is input and the position of the front corner P k It may be possible to recognize the position of the corners on the front of the moving object by inputting images of the moving object into a machine learning model that has been trained on images of multiple moving objects. Alternatively, the position of the front corner P may be recognized by inputting images of 701 moving objects using a 6D pose estimation technique using a convolutional neural network. k It is sufficient to recognize only that. 6D Pose is information that represents the position and orientation of an object using three rotation vectors and three translation vectors.
[0027] The search unit 204 is a part that has the function of searching for similar data to the image of the moving object 701, depending on the height of the holding unit 102 at the time of imaging and the position of the front corner of the recognized moving object 701. Similar data refers to images of the moving object 701 that have been stored in advance. The search unit 204 stores similar data captured under many conditions. The search unit 204 searches for similar data based on the height of the holding unit 102 at the time of imaging. The search unit 204 is a trained machine learning model that has stored multiple combinations of the front corner positions and rear corner positions of the captured moving object 701 as training datasets.
[0028] The estimation unit 205 is a part that has the function of estimating the position of the corners on the rear surface of the moving object according to the searched similar data. That is, when the position of the corners on the front surface of the moving object 701 is input to the estimation unit 205, the estimation unit 205 estimates the position of the corners on the rear surface of the moving object 701 using a trained machine learning model or the like that outputs the position of the corners on the rear surface of the moving object 701.
[0029] The object identification unit 206 is a part that has the function of identifying the state of the moving object 701 according to the position of the corners on the front of the recognized moving object 701 and the position of the corners on the rear of the moving object 701 that are estimated to be located there. For example, the object identification unit 206 identifies the three-dimensional attitude and position (6D pose) of the moving object 701 by solving a PnP problem based on the position of the corners on the front of the recognized moving object 701 and the position of the corners on the rear of the moving object 701. In order to solve the PnP problem, the internal parameters of the imaging device 103 are known. The size of the moving object 701 is also known. The PnP problem can be solved using known methods.
[0030] In the description of the embodiment, the terms "holding unit 102," "image acquisition unit 201," "height acquisition unit 202," "recognition unit 203," "search unit 204," "estimation unit 205," and "target identification unit 206" are used, but these may be replaced with "holding means," "acquisition means," "recognition means," "search means," "estimation means," and "target identification means."
[0031] The method of solving a PnP problem to determine the 6D pose from nine points—the positions of the front and rear corners and the center position—had the potential to fail to accurately estimate the 6D pose when large loads were loaded and the rear of the pallet was not visible in the image, or when the training was insufficient. By using the moving object identification system disclosed herein, the position and orientation of the object can be estimated with higher accuracy than techniques that estimate the 6D pose of the object based on a convolutional neural network, even when the rear of the moving object is not visible in the image or when the learning of the moving object is insufficient. Therefore, it can assist the operator of the mobile object 101. Furthermore, it can contribute to the realization of autonomous driving of the mobile object 101.
[0032] When estimating the position and orientation of a pallet using only barbed back (BB) data, it is necessary to estimate the corner positions using some method after acquiring the BB data. Therefore, the calculation may not be stable. However, the moving object identification system according to this embodiment can accurately identify the positions of corners on the rear surface in a short time, thus enabling stable determination of the position and orientation.
[0033] (Description of the method for identifying the target in the embodiment) Figure 3 is a flowchart of the target identification method according to the embodiment. The target identification method according to the embodiment will be explained with reference to Figure 3.
[0034] As shown in Figure 3, first an image is captured (step S301). An image of the target is captured using the imaging device 103. Next, the height of the imaging device is obtained (step S302). The information processing device 105 obtains the height of the imaging device 103 at the time of imaging. Next, the position of the front corners is recognized (step S303). The information processing device 105 recognizes the position of the corners present on the front of the imaged target. Next, similar data is searched for (step S304). The information processing device 105 searches for similar data that is similar to the image of the target, according to the height of the imaging device at the time of imaging and the position of the corners present on the front of the recognized target.
[0035] Next, the position of the rear corner is estimated (step S305). The information processing device 105 estimates the position of the corners on the rear of the object according to the searched similar data. Next, the object is identified using the positions of the front corners and the rear corners (step S306). The information processing device 105 identifies the state of the object according to the position of the corners on the front of the recognized object and the estimated position of the corners on the rear of the object.
[0036] By using this object identification method, the position and orientation of an object can be determined more efficiently than related technologies and with greater accuracy than methods that calculate the 6D pose from the position of corners on the front.
[0037] (Description of the moving object identification system according to Embodiment 1) Embodiment 1 is an example of an embodiment and includes examples of configurations and operations that are not essential. Figure 4 is a block diagram of the moving target identification system according to Embodiment 1. Figure 5 is a diagram showing the hierarchy of the dictionary data section according to Embodiment 1. The moving target identification system according to Embodiment 1 will be described with reference to Figures 4 and 5.
[0038] As shown in Figure 4, the moving target identification system 400 according to Embodiment 1 differs from the moving target identification system 100 according to Embodiment 1 in that it further includes a moving body 101 and a dictionary data unit 401.
[0039] The dictionary data unit 401 registers multiple similar data. These similar data are images of the moving object 701. The dictionary data unit 401 registers a large number of similar data taken from various heights and angles.
[0040] As shown in Figure 5, the dictionary data unit 401 has a hierarchical structure. The dictionary data unit 401 is connected to multiple cameras C attached to the holding unit 102. id For each of (id=1,2,··), the height H of the holding part 102 is i Similar data captured at (i=1,2,··) is saved. The similar data is point cloud data D, which includes the positions of various front corners P1(u,v), P2(u,v), P5(u,v), P6(u,v) and the positions of corners P3(u,v), P4(u,v), P7(u,v), P8(u,v) on the back surface. i (i=1,2,··) will be registered.
[0041] The exploration unit 204 identifies similar data with a small error in the height of the holding unit 102 when imaging and a small error in the position of the corner existing on the front surface of the moving object 701, from among the dictionary data in the dictionary data unit 401, using the same camera ID as the camera that imaged the moving object 701. Thereby, the exploration unit 204 searches for similar data similar to the image of the moving object 701. The similar data searched by the exploration unit 204 is preferably similar data with the smallest error in the height of the holding unit 102 when imaging and the smallest error in the position of the corner existing on the front surface of the moving object 701.
[0042] If similar data is discovered and completely matches the image of the moving object, the position of the corner existing on the rear surface of the similar data is used. However, there is little similar data that completely matches the image of the moving object. In that case, the position of the corner existing on the rear surface is estimated by the following calculation method.
[0043] First, a conversion formula λ k from a reference point such as P1 to a virtual point P k is calculated. λ k is represented by the following formula.
Equation
Equation
[0044] In this way, the position of the corner existing on the rear surface, which is a virtual point, is estimated from the position of the corner existing on the front surface. Then, by solving the PnP problem using the position of the corner existing on the front surface and the position of the corner existing on the rear surface, 6D Pose can be estimated with high accuracy.
[0045] By doing so, the position and orientation of the pallet can be efficiently estimated.
[0046] (Description of the method for identifying the target according to Embodiment 1) Figure 6 is a flowchart of the object identification method according to Embodiment 1. Figure 7 is a diagram showing the recognition of the front corner position, dictionary data search, and estimation of the rear corner position according to Embodiment 1. Figure 8 is a diagram showing how the position and orientation of the moving object are determined from the front and rear corner positions by solving a PnP problem according to Embodiment 1. The object identification method according to Embodiment 1 will be explained with reference to Figures 6 to 8.
[0047] As shown in Figure 6, first an image is acquired (step S601). The imaging device 103 captures an image of the moving object 701. Next, the information processing device 105 acquires the imaging device ID (Identification), imaging device internal parameters, and imaging device height (step S602). The internal parameters differ for each imaging device 103. The internal parameters of the imaging device 103 are necessary when solving the PnP problem. Therefore, the information processing device 105 needs to acquire the imaging device ID and its internal parameters.
[0048] Next, the information processing device 105 recognizes the position of the front corner of the object being moved (step S603). Next, the information processing device 105 searches for matching data from the dictionary data unit 401 (step S604). Next, the information processing device 105 estimates the position of the rear corner of the object being moved 701 (step S605). These three processes will be explained with reference to Figure 7.
[0049] As shown in the upper left of Figure 7, the positions of four corners P1, P2, P5, and P6 on the front of the moving object 701 are recognized using machine learning or the like. Next, as shown in the lower part of Figure 7, similar data is searched from the dictionary data unit 401. Similar data is found that matches the imaging device 103, has the smallest error in the height of the imaging device, and matches the position of the frontmost corner in the captured image, and the positions of four corners P3, P4, P7, and P8 on the rear surface are estimated as shown in the upper left of Figure 7. The estimation method is as described above, using a virtual point P, which is the position of the corner on the rear surface, from the reference point P1. kConversion formula λ k Calculate and convert from reference point P1 to virtual point P k This is a method for calculating it.
[0050] Finally, the information processing device 105 solves the PnP problem using the positions of the front corners and the rear corners to determine the position and orientation (step S606). As shown in Figure 8, the information processing device 105 determines the position of the corners of the moving object 701, P k (u k ,v k We solve the PnP problem using (k=0,1···). Here, P0(u0,v0) is the center point of the moving object 701. Solving the PnP problem gives us R|t for the 6D pose, which is the 3D orientation of the moving object 701.
[0051] By estimating the 6D pose, it becomes possible to calculate what turning radius and how many degrees the mobile body 101 needs to rotate to be able to face the moving object 701 directly.
[0052] It is also possible to estimate the 6D pose of the pallet geometrically by 3D reconstructing the front of the pallet using a depth sensor without solving the PnP problem. However, this would require a high-precision depth sensor and stable acquisition of distance information such as LiDAR. Therefore, high-precision estimation is difficult with inexpensive RGB-D cameras. As described above, by solving the PnP problem using the positions of the front and rear corners to determine the position and orientation, it is possible to estimate the position and orientation of a moving object using an inexpensive RGB-D camera.
[0053] As disclosed in Embodiments and Embodiment 1, the position and orientation of a moving object can be estimated with high accuracy by estimating the position of the corners on the rear surface of the moving object according to the position of the corners on the front surface of the moving object.
[0054] Furthermore, some or all of the processing in the information processing device 105 described above can be implemented as a computer program. Such a program can be stored using various types of non-temporary computer-readable media and supplied to a computer. Non-temporary computer-readable media include various types of tangible recording media. Examples of non-temporary computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). Programs may also be supplied to a computer using various types of temporary computer-readable media. Examples of temporary computer-readable media include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable media can be supplied to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.
[0055] It should be noted that the present invention is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention.
[0056] Some or all of the above embodiments may be described as follows, but are not limited to the following. (Note 1) A holding means for holding the object to be moved, A height acquisition means for acquiring the height of the imaging device attached to the holding means, A recognition means for recognizing the position of a corner on the front surface of the moving object captured using the aforementioned imaging device, A search means for searching for similar data to the image of the moving object, depending on the height of the imaging device acquired and the position of the corner present on the front of the recognized moving object, Estimation means for estimating the position of the corner present on the rear surface of the moving object according to the similar data that has been searched, A means for identifying the state of the moving object according to the position of the corners on the front of the recognized moving object and the position of the corners on the rear of the moving object estimated, A system for identifying moving targets. (Note 2) The recognition means recognizes the position of a corner present on the front of the moving object according to the amount of change in features in the image, as described in Appendix 1 of the moving object identification system. (Note 3) The recognition means is a moving object identification system as described in Appendix 1, in which an image of the moving object is input to a machine learning model that has learned images of multiple moving objects, and the position of the corner present on the front of the moving object is recognized. (Note 4) The aforementioned similar data is registered multiple times in the dictionary data. The moving target identification system according to Appendix 1, wherein the search means searches for similar data that is similar to the image of the moving target by identifying similar data obtained from the dictionary data in which the height of the imaging device is similar and the error in the position of the corners present on the front of the moving target is small. (Note 5) The state of the moving object is the three-dimensional orientation and position of the moving object, as described in Appendix 1, for the moving object identification system. (Note 6) The object to be moved is rectangular in shape. The number of corner positions on the front surface of the moving object is four. The moving object identification system described in Appendix 1, wherein the number of corner positions on the rear surface of the moving object is four. (Note 7) The moving object identification system according to Appendix 1, wherein the holding means is a fork, the moving object is a pallet of a fixed size, and the imaging device is an RGB-D camera. (Note 8) The moving object identification system according to Appendix 5, wherein the identification means identifies the three-dimensional orientation and position of the moving object by solving a PnP problem using the position of the corner present on the front of the recognized moving object and the position of the corner present on the rear of the moving object. (Note 9) A holding means for holding the object to be moved, A height acquisition means for acquiring the height of the imaging device attached to the holding means, A recognition means for recognizing the position of a corner on the front surface of the moving object captured using the aforementioned imaging device, A search means for searching for similar data to the image of the moving object, depending on the height of the imaging device acquired and the position of the corner present on the front of the recognized moving object, Estimation means for estimating the position of the corner present on the rear surface of the moving object according to the similar data that has been searched, A means for identifying the state of the moving object according to the position of the corners on the front of the recognized moving object and the position of the corners on the rear of the moving object estimated, A device for identifying moving targets, equipped with the following features. (Note 10) The recognition means recognizes the position of a corner present on the front of the moving object according to the amount of change in features in the image, as described in Appendix 9. (Note 11) The recognition means is a moving object identification device as described in Appendix 9, which inputs an image of the moving object to a machine learning model that has learned images of multiple moving objects and recognizes the position of a corner present on the front of the moving object. (Note 12) The aforementioned similar data is registered multiple times in the dictionary data. The moving target identification device according to Appendix 9, wherein the search means searches for similar data that is similar to the image of the moving target by identifying similar data obtained from the dictionary data in which the height of the imaging device is similar and the error in the position of the corners present on the front of the moving target is small. (Note 13) The state of the moving object is the three-dimensional orientation and position of the moving object, as described in Appendix 9 of the moving object identification device. (Note 14) The object to be moved is rectangular in shape. The number of corner positions on the front surface of the moving object is four. The moving object identification device described in Appendix 9, wherein the number of corner positions on the rear surface of the moving object is four. (Note 15) The moving object identification device according to Appendix 9, wherein the holding means is a fork, the moving object is a pallet of a fixed size, and the imaging device is an RGB-D camera. (Note 16) The moving object identification device according to Appendix 13, wherein the identification means identifies the three-dimensional orientation and position of the moving object by solving a PnP problem using the position of a corner present on the front of the recognized moving object and the position of a corner present on the rear of the moving object that has been estimated. (Note 17) Using an imaging device, capture an image of the target, The height of the imaging device is obtained, The position of the corners on the front surface of the object captured using the imaging device is recognized. Depending on the height of the imaging device obtained and the position of the corners on the front of the recognized object, similar data similar to the image of the object is searched for. Based on the similar data that was searched, the position of the corner present on the rear surface of the object is estimated. A method for identifying an object, which identifies the state of the object according to the position of the corners on the front of the recognized object and the estimated position of the corners on the rear of the object. (Note 18) The aforementioned recognition is the object identification method described in Appendix 17, which recognizes the position of the corner present on the front of the object according to the amount of change in the features in the image. (Note 19) The aforementioned recognition is the object identification method described in Appendix 17, in which an image of the object is input to a machine learning model that has been trained on images of multiple objects, and the position of the corner present on the front of the object is recognized. (Note 20) The aforementioned similar data is registered multiple times in the dictionary data. The method for identifying an object as described in Appendix 17, wherein the search involves identifying similar data from the dictionary data that have similar heights of the imaging device and small errors in the position of corners present on the front of the object, thereby searching for similar data that are similar to the image of the object. (Note 21) The object identification method described in Appendix 17, wherein the state of the object is the three-dimensional orientation and position of the object. (Note 22) The aforementioned object is rectangular in shape, The number of corner positions on the front surface of the aforementioned object is four. The method for identifying the target as described in Appendix 17, wherein the number of corner positions on the rear surface of the target is four. (Note 23) The object identification method described in Appendix 17, wherein the object is a pallet of a fixed size, and the imaging device is an RGB-D camera. (Note 24) The object identification method described in Appendix 21, wherein the identification is determined by solving a PnP problem using the positions of the corners on the front of the recognized object and the estimated positions of the corners on the rear of the object, thereby determining the three-dimensional orientation and position of the object. (Note 25) Using an imaging device, capture an image of the target, The height of the imaging device is obtained, The position of the corner present on the front surface of the object captured using the aforementioned imaging device is recognized. Depending on the height of the imaging device obtained and the position of the corners on the front of the recognized object, similar data similar to the image of the object is searched for. Based on the similar data that was searched, the position of the corner present on the rear surface of the object is estimated. A non-temporary computer-readable medium storing a program that causes an information processing device to perform the operation of identifying the state of the object according to the position of the corners on the front of the recognized object and the estimated position of the corners on the rear of the object. (Note 26) The aforementioned recognition is performed on a non-temporary computer-readable medium that stores the program described in Appendix 25, which recognizes the position of a corner present on the front of the object according to the amount of change in the features in the image. (Note 27) The aforementioned recognition is performed by storing a non-temporary computer-readable medium that stores the program described in Appendix 25, which inputs an image of a target into a machine learning model that has been trained on images of multiple targets, and recognizes the position of the corners present on the front of the target. (Note 28) The aforementioned similar data is registered multiple times in the dictionary data. A non-temporary computer-readable medium storing the program described in Appendix 25, which searches for similar data that is similar to the image of the target by identifying similar data from the dictionary data in which the height of the imaging device is similar and the error in the position of the corners present on the front of the target is small. (Note 29) The state of the object is the three-dimensional orientation and position of the object, and the non-temporary computer-readable medium storing the program described in Appendix 25. (Note 30) The aforementioned object is rectangular in shape, The number of corner positions on the front surface of the aforementioned object is four. A non-temporary computer-readable medium for storing the program described in Appendix 25, wherein the number of corner positions on the rear surface of the aforementioned object is four. (Note 31) A non-temporary computer-readable medium for storing the program according to claim 25, wherein the object is a pallet of a fixed size, and the imaging device is an RGB-D camera. (Note 32) The aforementioned identification is a non-temporary computer-readable medium storing the program described in Appendix 29, which identifies the three-dimensional orientation and position of the object by solving a PnP problem using the positions of the corners on the front of the recognized object and the estimated positions of the corners on the rear of the object. [Explanation of Symbols]
[0057] 100 Moving object identification system, 101 Moving object, 102 Holding unit, 103 Imaging device, 104 Sensor, 105 Information processing device, 201 Image acquisition unit, 202 Height acquisition unit, 203 Recognition unit, 204 Search unit, 205 Estimation unit, 206 Object identification unit, 400 Moving object identification system, 401 Dictionary data unit, 701 Moving object
Claims
1. A holding means for holding the object to be moved, A height acquisition means for acquiring the height of the imaging device attached to the holding means, A recognition means for recognizing the position of a corner on the front surface of the moving object captured using the aforementioned imaging device, A search means for searching for similar data to the image of the moving object, according to the height of the imaging device acquired and the position of the corner present on the front of the recognized moving object, Estimation means for estimating the position of the corner present on the rear surface of the moving object according to the similar data that has been searched, A means for identifying the state of the moving object according to the position of the corners on the front of the recognized moving object and the position of the corners on the rear of the moving object estimated, A system for identifying moving targets.
2. The moving object identification system according to claim 1, wherein the recognition means recognizes the position of a corner present on the front of the moving object according to the amount of change in features in the image.
3. The moving object identification system according to claim 1, wherein the recognition means is input to a machine learning model that has learned images of multiple moving objects, and recognizes the position of the corners present on the front of the moving object.
4. The aforementioned similar data is registered multiple times in the dictionary data. The moving target identification system according to claim 1, wherein the search means searches for similar data that is similar to the image of the moving target by identifying similar data from the dictionary data in which the height of the acquired imaging device is similar and the error in the position of the corner present on the front of the moving target is small.
5. The moving object identification system according to claim 1, wherein the state of the moving object is the three-dimensional orientation and position of the moving object.
6. The object to be moved is rectangular in shape. The number of corner positions on the front surface of the moving object is four. The moving object identification system according to claim 1, wherein the number of corner positions on the rear surface of the moving object is four.
7. The moving object identification system according to claim 1, wherein the holding means is a fork, the moving object is a pallet of a fixed size, and the imaging device is an RGB-D camera.
8. A holding means for holding the object to be moved, A height acquisition means for acquiring the height of the imaging device attached to the holding means, A recognition means for recognizing the position of a corner on the front surface of the moving object captured using the aforementioned imaging device, A search means for searching for similar data to the image of the moving object, according to the height of the imaging device acquired and the position of the corner present on the front of the recognized moving object, Estimation means for estimating the position of the corner present on the rear surface of the moving object according to the similar data that has been searched, A means for identifying the state of the moving object according to the position of the corners on the front of the recognized moving object and the position of the corners on the rear of the moving object estimated, A device for identifying moving targets, equipped with the following features.
9. Using an imaging device, capture an image of the target, The height of the imaging device is obtained, The position of the corners on the front surface of the object captured using the imaging device is recognized. Based on the height of the imaging device obtained and the position of the corners on the front of the recognized object, similar data similar to the image of the object is searched for. Based on the similar data that was searched, the position of the corner present on the rear surface of the object is estimated. A method for identifying an object, which identifies the state of the object according to the position of the corners on the front of the recognized object and the estimated position of the corners on the rear of the object.
10. Using an imaging device, capture an image of the target, The height of the imaging device is obtained, The position of the corner present on the front surface of the object captured using the aforementioned imaging device is recognized. Depending on the height of the imaging device obtained and the position of the corners on the front of the recognized object, similar data similar to the image of the object is searched for. Based on the similar data that was searched, the position of the corner present on the rear surface of the object is estimated. A program that causes an information processing device to determine the state of the object according to the position of the corners on the front of the recognized object and the estimated position of the corners on the rear of the object.