Position and orientation estimation device, position and orientation estimation method, and position and orientation estimation program

Abstract

Provided are a position and orientation estimation device, a position and orientation estimation method, and a position and orientation estimation program which each make it possible to estimate the position and orientation of an object at lower cost and with higher accuracy. A position and orientation estimation device according to the present invention comprises: an RGB camera that is mounted in a forklift and acquires an RGB image; a front face information estimation means that estimates, from the RGB image, front face information which pertains to the front face of a pallet; an inference means that uses a regression model which has been trained using, as training data, front face information and position-and-orientation-related information pertaining to the position and orientation of a pallet so as to infer position-and-orientation-related information from the front face information estimated by the front face information estimation means and to calculate the reliability of the inferred position-and-orientation-related information; and a removal means that removes the position-and-orientation-related information as an outlier if the reliability calculated by the inference means is not more than a prescribed threshold value.

Description

Position and orientation estimation device, position and orientation estimation method, and position and orientation estimation program

[0001] The present disclosure relates to a position and orientation estimation device, a position and orientation estimation method, and a position and orientation estimation program.

[0002] When automating a forklift, it is necessary to estimate the position and orientation of a pallet, which is a loading object, with high accuracy. Patent Document 1 describes a technology for estimating the position and orientation of a pallet by attaching a two-dimensional lidar (2D LiDAR) to a movable part of a linear actuator and moving the 2D LiDAR up and down to reconstruct the front of the pallet in three dimensions.

[0003] Japanese Patent Application Laid-Open No. 2022-034408

[0004] However, the technique in Patent Document 1 requires a two-dimensional lidar, which is a highly accurate distance sensor, which results in an expensive position and orientation estimation device. Furthermore, when using a two-dimensional lidar, distance measurement must be performed at a distance of, for example, about 2 m from the pallet, which results in a problem that the position and orientation of the pallet cannot be estimated from a distance greater than that.

[0005] Therefore, costs can be reduced by estimating the position and orientation of the pallet from RGB-D images acquired with a relatively inexpensive RGB-D camera. However, if only the front of the pallet is visible in the RGB-D image, or if cargo is placed on the pallet, the estimation accuracy decreases. Therefore, in order to estimate the position and orientation of the pallet from the RGB-D image with high accuracy, it is necessary to attach AR markers to the pallet. Attaching AR markers to all pallets for forklift automation would be expensive.

[0006] An object of the present disclosure is to provide a position and orientation estimation device, a position and orientation estimation method, and a position and orientation estimation program that are capable of estimating the position and orientation of an object at lower cost and with higher accuracy.

[0007] A position and orientation estimation device according to the present disclosure is mounted on a forklift and includes an image acquisition unit that acquires an RGB image; a front surface information estimation unit that estimates front surface information, which is information relating to the front surface of an object, from the RGB image; an inference unit that infers the position and orientation related information from the front surface information estimated by the front surface information estimation unit using a trained regression model that uses the front surface information and position and orientation related information, which is information relating to the position and orientation of the object, as training data, and calculates reliability of the inferred position and orientation related information; and a removal unit that removes the position and orientation related information as an outlier if the reliability calculated by the inference unit is equal to or less than a predetermined threshold.

[0008] The position and orientation estimation method according to the present disclosure is a method in which a computer estimates front surface information, which is information relating to the front surface of an object, from an RGB image captured by an RGB camera mounted on a forklift, infers the position and orientation related information from the estimated front surface information using a trained regression model that uses the front surface information and position and orientation related information, which is information relating to the position and orientation of the object, as training data, calculates the reliability of the inferred position and orientation related information, and removes the position and orientation related information as an outlier if the calculated reliability is equal to or less than a predetermined threshold.

[0009] A position and orientation estimation program according to the present disclosure is a program that causes a computer to execute the following processes: a process of estimating front surface information, which is information relating to the front surface of an object, from an RGB image captured by an RGB camera mounted on a forklift; a process of inferring the position and orientation related information from the estimated front surface information using a regression model that has been trained using the front surface information and position and orientation related information, which is information relating to the position and orientation of the object, as training data, and calculating the reliability of the inferred position and orientation related information; and a process of removing the position and orientation related information as an outlier if the calculated reliability is equal to or less than a predetermined threshold.

[0010] According to the present disclosure, it is possible to provide a position and orientation estimation device, a position and orientation estimation method, and a position and orientation estimation program that are capable of estimating the position and orientation of an object at lower cost and with higher accuracy.

[0011] FIG. 1 is a schematic block diagram showing an example of the configuration of a position and orientation estimation device according to the present disclosure. FIG. 2 is a diagram explaining an example of pallet front surface information according to the present disclosure. FIG. 3 is a flowchart showing an example of a position and orientation estimation method according to the present disclosure. FIG. 4 is a schematic block diagram showing another example of the configuration of a position and orientation estimation device according to the present disclosure. FIG. 5 is a diagram explaining an example of teacher data according to the present disclosure. FIG. 6 is a diagram explaining another example of pallet front surface information according to the present disclosure. FIG. 7 is a diagram explaining an example of estimation processing of pallet front surface information according to the present disclosure. FIG. 8 is a flowchart showing another example of a position and orientation estimation method according to the present disclosure. FIG. 9 is a block diagram showing an example of the configuration of a computer according to the present disclosure.

[0012] Embodiment 1 An example of the configuration of a position and orientation estimation device 10 according to the present disclosure will be described below with reference to Fig. 1. The position and orientation estimation device 10 estimates the position and orientation of a pallet, which is an object to be handled by a forklift (not shown), based on an RGB image. Specifically, the position and orientation estimation device 10 includes an RGB camera 11 as an image acquisition unit, a front surface information estimation unit 12, an inference unit 13, and a removal unit 14.

[0013] The RGB camera 11 is mounted on the forklift and captures an image of a predetermined area around the forklift to obtain an RGB image. Figure 2 shows an example of an RGB image captured by the RGB camera 11. The RGB image shown in Figure 2 shows a pallet 200 loaded with packages 301, 302, 303, and 304. The pallet 200 has holes 200A and 200B into which the forks of the forklift are inserted.

[0014] The front surface information estimation unit 12 estimates front surface information, which is information relating to the front surface of the pallet 200, from the RGB image acquired by the RGB camera 11. Here, the front surface information is, for example, the image coordinates of four corner points P1 to P4 on the front surface of the pallet 200 as shown in Figure 2. Note that the front surface information is not limited to the image coordinates of the four corner points P1 to P4.

[0015] The inference unit 13 uses a trained regression model to infer position and orientation related information, which is information about the position and orientation of the pallet 200, from the front surface information estimated by the front surface information estimation unit 12.

[0016] The regression model is trained using front surface information and position-and-posture related information as training data. Specifically, in training the regression model, the front surface information is input to the regression model, and regression parameters (weights) of the regression model are calculated so that information output from the regression model approximates the position-and-posture related information of the training data.

[0017] Furthermore, the position and orientation related information is, for example, information about the rear surface of the pallet 200 shown in FIG. 2. Specifically, the position and orientation related information is the image coordinates of the four corner points of the rear surface of the pallet 200. In the RGB image shown in FIG. 2, the four corner points of the rear surface of the pallet 200 are hidden by the luggage 301, 302, 303, and 304. Therefore, the inference unit 13 infers the image coordinates of the four corner points of the rear surface of the pallet 200 as the position and orientation related information. Then, it becomes possible to estimate the position and orientation of the pallet based on the front surface information of the pallet 200 (image coordinates of the four corner points of the front surface) and the position and orientation related information (image coordinates of the four corner points of the rear surface). Note that the position and orientation related information is not limited to the image coordinates of the four corner points of the rear surface of the pallet 200.

[0018] The inference unit 13 also estimates position-posture related information of the pallet 200 and calculates the reliability of the inferred position-posture related information.

[0019] If the reliability calculated by the inference unit 13 is equal to or less than a predetermined threshold, the removal unit 14 removes the position and orientation related information inferred by the inference unit 13 as an outlier. This makes it possible to prevent position and orientation related information with low reliability from being used in the process of estimating the position and orientation of the pallet 200.

[0020] 3 is a flowchart illustrating an example of a position and orientation estimation method according to the present disclosure. First, the RGB camera 11 captures an image of a predetermined area around the forklift to acquire an RGB image (step S11). Next, the front surface information estimation unit 12 estimates front surface information, which is information about the front surface of the pallet 200, from the RGB image acquired by the RGB camera 11 (step S12). Next, the inference unit 13 infers position and orientation related information of the pallet 200 from the front surface information estimated by the front surface information estimation unit 12 using a trained regression model and calculates reliability (step S13). Next, the removal unit 14 determines whether the reliability calculated in step S13 is equal to or less than a predetermined threshold (step S14). If the reliability is equal to or less than the predetermined threshold in step S14 (step S14; Yes), the removal unit 14 removes the position and orientation related information of the pallet 200 inferred in step S13 (step S15), and the process ends.

[0021] In the position and orientation estimation device 10 according to the present disclosure, front surface information of the pallet 200 is estimated from an RGB image, and position and orientation related information of the pallet 200 is inferred from the estimated front surface information using a trained regression model. Therefore, the position and orientation of the pallet 200 can be estimated using a relatively inexpensive RGB camera 11, without requiring a 2D LIDAR, which is a highly accurate distance sensor. Furthermore, the removal unit 14 removes position and orientation related information with low reliability as an outlier, allowing the position and orientation of the pallet 200 to be estimated with high accuracy. Therefore, there is no need to attach AR markers to the pallets 200, and the cost of attaching AR markers to all pallets 200 can be saved. Therefore, the position and orientation of the pallet 200 can be estimated more inexpensively and with high accuracy.

[0022] Embodiment 2 Next, with reference to FIG. 4 , another example of the configuration of the position and orientation estimation apparatus 100 according to the present disclosure will be described. As shown in FIG. 4 , the position and orientation estimation apparatus 100 includes a pre-processing unit 110 and a real-time processing unit 120. The pre-processing unit 110 includes an RGB camera 121, a camera position and orientation calculation unit 111, a palette position and orientation calculation unit 112, a teacher data generation unit 113, a learning unit 114, a model storage unit 130, etc. The real-time processing unit 120 includes the RGB camera 121, a front surface information estimation unit 122, an inference unit 123, a removal unit 124, a position and orientation estimation unit 125, a model storage unit 130, etc. In other words, the pre-processing unit 110 and the real-time processing unit 120 share the RGB camera 121 and the model storage unit 130. The model storage unit 130 stores a regression model. In the position and orientation estimation apparatus 100, the pre-processing unit 110 previously learns the regression model. Then, in the real-time processing unit 120, the position and orientation of the pallet 200 are estimated from the RGB image using the trained regression model. The pre-processing unit 110 and the real-time processing unit 120 will be described in detail below.

[0023] <Configuration and Processing of Preprocessing Unit 110> The RGB camera 121 is mounted at a predetermined position on the forklift. The RGB camera 121 captures an image of a reference pallet 201, which serves as a reference, and acquires a reference RGB image for generating training data. FIG. 5 shows an example of a reference RGB image acquired by the RGB camera 121. The reference RGB image shown in FIG. 5 captures the reference pallet 201 without any cargo loaded. Note that the reference RGB image may also be an image of the reference pallet 201 loaded with cargo. The reference pallet 201 has the same shape and size as the pallet 200. As shown in FIG. 5, an AR marker M is attached to the reference pallet 201. The reference pallet 201 is placed at a predetermined position, and the RGB camera 121 captures images of the reference pallet 201 from various positions. That is, the RGB camera 121 acquires multiple reference RGB images.

[0024] The camera position and orientation calculation unit 111 calculates, from the reference RGB image, the position and orientation of the RGB camera 121. The camera position and orientation calculation unit 111 calculates, from the reference RGB image, the position and orientation of the RGB camera 121 using, for example, Visual SLAM. Specifically, the position and orientation of the RGB camera 121 calculated by the camera position and orientation calculation unit 111 are expressed in coordinates based on the RGB camera 121 (hereinafter referred to as "camera coordinates").

[0025] The palette position and orientation calculation unit 112 calculates the position and orientation of the reference palette 201 from the reference RGB image. For example, the palette position and orientation calculation unit 112 detects AR markers M from the reference RGB image, and calculates the position and orientation of the reference palette 201 based on information about the positions and orientations of the AR markers M. Specifically, the position and orientation of the reference palette 201 calculated by the palette position and orientation calculation unit 112 are expressed in world coordinates, for example, with the reference palette 201 as the reference.

[0026] The training data generation unit 113 generates training data based on the position and orientation of the RGB camera 121 calculated by the camera position and orientation calculation unit 111 and the position and orientation of the reference pallet 201 calculated by the pallet position and orientation calculation unit 112. The training data is a set of explanatory variables that serve as input to the regression model and a target variable that serves as the output of the regression model. The explanatory variables include, for example, the image coordinates of four corner points P1 to P4 on the front surface of the reference pallet 201 captured in the reference RGB image shown in FIG. 5 , the image coordinates of an upper left corner point P5 of a bounding box B1 that surrounds the entire reference pallet 201 and the width and height of the bounding box B1, the image coordinates of an upper left corner point P6 of a bounding box B2 that surrounds a hole 201A in the reference pallet 201 and the width and height of the bounding box B2, and the image coordinates of an upper left corner point P7 of a bounding box B3 that surrounds a hole 201B in the reference pallet 201 and the width and height of the bounding box B3. The objective variables also include, for example, the image coordinates of four corner points P8 to P11 of the rear surface of the reference pallet 201 captured in the reference RGB image shown in FIG.

[0027] Specifically, the training data generating unit 113 generates training data from an RGB image using a calculation formula for determining the position and orientation of an object in the RGB image. The calculation formula will be described below.

[0028] First, the camera coordinates (X ｃ , Y ｃ , Z ｃ ) and world coordinates (X ｗ , Y ｗ , Z ｗ ), the following equation (1) holds between the RGB image and the object, where t is a translation vector representing the position of the object in the RGB image and R is a rotation vector representing the orientation of the object. Furthermore, when an internal parameter of the RGB camera 121 is K, the following equation (2) holds between the image coordinates of the RGB image and the camera coordinates of the RGB camera 121. In equation (2), s(u, v) represents the depth at the image coordinate (u, v). From equations (1) and (2), equation (3) is established. When the equation (3) is expressed using a matrix, it is expressed as the following equation (4). In equations (3) and (4), the internal parameters of the RGB camera 121 are known values. The teacher data generation unit 113 can calculate (R|t) from the position and orientation of the RGB camera 121 calculated by the camera position and orientation calculation unit 111 and the position and orientation of the reference pallet 201 calculated by the pallet position and orientation calculation unit 112. Then, the position (X ｗ , Y ｗ , Z ｗ ) are known. Therefore, the training data generation unit 113 can calculate the image coordinates (u, v) of each of points P1 to P11 of the reference palette 201 using equation (3) or equation (4).

[0029] The learning unit 114 learns a regression model using the training data generated by the training data generation unit 113. The regression model is, for example, a kernel ridge regression model. The kernel ridge regression model is expressed by the following equation (5). In the above formula (5), (xｉ , x) are explanatory variables, f(x) is the target variable, α ｉ is a regression parameter, and k is a Gaussian kernel function. The Gaussian kernel function k(x, y) is expressed by the following equation (6). where β is an arbitrary parameter β>0, Is L ２ Norm and Here, K is called the Gram matrix, and is a real symmetric matrix K = K Ｔ Then, the residual sum of squares R(α) is The point where the residual sum of squares is differentiated and becomes 0 is the extreme value and minimum value, The regression parameter α is Here, if we add a regularization term to the residual sum of squares (ridge regression), we get where λ is the regularization parameter. The point where the residual sum of squares with the regularization term added becomes 0 when differentiated is the extreme minimum value, The regression parameter α is expressed by the following equation (7). However, I Ｎ is an N×N unit matrix. The learning unit 114 calculates the regression parameter α using the above formula (7) and the training data generated by the training data generating unit 113.

[0030] <Configuration and Processing of Real-Time Processing Unit 120> The RGB camera 121 captures an image of the pallet 200 and acquires an RGB image. Fig. 6 shows an example of an RGB image acquired by the RGB camera 121. The RGB image shown in Fig. 6 shows the pallet 200 loaded with packages 301, 302, 303, and 304. The shape and size of the pallet 200 are the same as those of the reference pallet 201. Furthermore, no AR marker M is attached to the pallet 200.

[0031] The front surface information estimation unit 122 estimates front surface information of the pallet 200 from the RGB image acquired by the RGB camera 121. Here, the front surface information includes, for example, the image coordinates of four corner points P1 to P4 on the front surface of the pallet 200 captured in the RGB image shown in Fig. 6, the image coordinates of an upper left corner point P5 of a bounding box B1 that surrounds the entire pallet 200 and the width and height of the bounding box B1, the image coordinates of an upper left corner point P6 of a bounding box B2 that surrounds a hole 200A in the pallet 200 and the width and height of the bounding box B2, and the image coordinates of an upper left corner point P7 of a bounding box B3 that surrounds a hole 200B in the pallet 200 and the width and height of the bounding box B3.

[0032] Specifically, the front surface information estimation unit 122 acquires front surface information of the palette 200 using YOLO (You Only Look Once). More specifically, as shown in FIG. 7 , the front surface information estimation unit 122 detects bounding boxes B1 to B3 using YOLO. Furthermore, the front surface information estimation unit 122 estimates areas A1 to A4 in which four corner points P1 to P4 exist on the front surface of the palette 200, based on the YOLO detection results. The front surface information estimation unit 122 then estimates points in the areas A1 to A4 that satisfy both the conditions of being "close to the center of the palette 200" and "starting points where edges are connected in the vertical and horizontal directions" as corner points P1 to P4.

[0033] The inference unit 123 uses a trained regression model to infer position and orientation related information, which is information related to the position and orientation of the pallet 200, from the front surface information estimated by the front surface information estimation unit 122. In addition to estimating the position and orientation related information of the pallet 200, the inference unit 123 calculates the reliability of the inferred position and orientation related information.

[0034] Specifically, the inference unit 123 inputs the front surface information estimated by the front surface information estimation unit 122 as an explanatory variable into equation (5), and calculates the position and orientation related information of the pallet 200 as the objective variable. Here, the position and orientation related information of the pallet 200 is, for example, the image coordinates of the four corner points of the rear surface of the pallet 200 (points corresponding to corner points P8 to P11 shown in FIG. 5). In the RGB image shown in FIG. 6, the four corner points of the rear surface of the pallet 200 are hidden by the luggage 301, 302, 303, and 304. Therefore, the inference unit 123 infers the image coordinates of the four corner points of the rear surface of the pallet 200 as the position and orientation related information.

[0035] The inference unit 123 also calculates the value of the Gaussian kernel function k(x, y) expressed by equation (6) as the reliability of the inferred position-posture related information. The value (reliability) of the Gaussian kernel function k(x, y) expressed by equation (6) ranges from 0 to 1, and the closer to 1 the value is, the higher the reliability of the inferred position-posture related information.

[0036] If the reliability calculated by the inference unit 123 is equal to or less than a predetermined threshold, the removal unit 124 removes the position and orientation related information inferred by the inference unit 123 as an outlier. This prevents position and orientation related information with low reliability from being used in the process of estimating the position and orientation of the pallet 200.

[0037] The position and orientation estimation unit 125 estimates the position and orientation of the pallet 200 based on the front surface information estimated by the front surface information estimation unit 122 and the position and orientation associated information inferred by the inference unit 123. Specifically, the position and orientation estimation unit 125 estimates the position and orientation of the pallet 200 based on the position and orientation associated information that was not removed by the removal unit 124 and the front surface information that was used as an explanatory variable when inferring the position and orientation associated information. More specifically, the position and orientation estimation unit 125 calculates a translation vector t that represents the position of the pallet 200 and a rotation vector R that represents the orientation of the pallet 200 using equation (3) or equation (4). As described above, in equations (3) and (4), the internal parameters of the RGB camera 121 are known values. For this reason, the position and orientation estimation unit 125 inputs, for example, the image coordinates (u, v) of the upper left corner point P1 and the upper right corner point P2 of the front surface of the pallet 200 as front surface information of the pallet 200, and the image coordinates (u, v) of the upper left corner point (a point corresponding to corner point P8 shown in FIG. 5 ) and the upper right corner point (a point corresponding to corner point P9 shown in FIG. 5 ) of the rear surface of the pallet 200 as position and orientation related information, into equation (3) or equation (4), and calculates the translation vector t and rotation vector R of the pallet 200. The position and orientation estimation unit 125 calculates the translation vector t and rotation vector R of the pallet 200 using, for example, a PnP (Perspective-n-Point) solver.

[0038] Next, an example of a position and orientation estimation method according to the present disclosure will be described with reference to Figure 8. The processing of steps S101 to S105 shown in Figure 8 is similar to steps S11 to S15 shown in Figure 3, and therefore description thereof will be omitted. Next, the position and orientation estimation unit 125 estimates the position and orientation of the pallet 200 using the front surface information estimated in step S102 and the position and orientation related information inferred in step S103 (step S106), and this processing ends.

[0039] In the position and orientation estimation device 100 according to the present disclosure, front surface information of the pallet 200 is estimated from an RGB image, and position and orientation related information of the pallet 200 is inferred from the estimated front surface information using a trained kernel ridge regression model. Therefore, the position and orientation of the pallet 200 can be estimated using a relatively inexpensive RGB camera 11, without requiring a 2D LIDAR, which is a highly accurate distance sensor. Furthermore, the removal unit 124 removes position and orientation related information with low reliability as an outlier, allowing the position and orientation of the pallet 200 to be estimated with high accuracy. Therefore, there is no need to attach AR markers to the pallets 200, and the cost of attaching AR markers to all pallets 200 can be saved. Therefore, the position and orientation of the pallet 200 can be estimated more inexpensively and with high accuracy.

[0040] Furthermore, the position and orientation related information is the image coordinates of the four corner points of the rear surface of the pallet 200, and the position and orientation estimation unit 125 estimates the position and orientation of the pallet 200 based on the position and orientation related information and the front surface information. This allows the position and orientation of the pallet 200 to be estimated, enabling automatic operation of the forklift.

[0041] Furthermore, since the front surface information estimation unit 122 estimates the front surface information of the pallet 200 using YOLO, it is possible to estimate the front surface information of the pallet 200 without using an AR marker. Therefore, it is possible to save the cost of attaching AR markers to all pallets 200.

[0042] Furthermore, by using a kernel ridge regression model as the regression model, it becomes possible to remove outliers by using the value of the Gaussian kernel function as the reliability.

[0043] In the above-described embodiment, the present invention has been described as a hardware configuration, but the present invention is not limited to this. The above-described functions (processing) of the position and orientation estimation apparatus 10, 100 may be realized by a computer 400 having the following configuration, for example.

[0044] 9 is a block diagram showing the configuration of a computer 400 that realizes the processing of the position and orientation estimation apparatuses 10 and 100. As shown in FIG. 9, the computer 400 includes a memory 401 and a processor 402.

[0045] The memory 401 is configured, for example, by a combination of a volatile memory and a non-volatile memory. The memory 401 is used to store programs executed by the processor 402, data used for various processes, and the like. The model storage unit 130 of the position and orientation estimation apparatus 100 may be realized by the memory 401. However, these may also be realized by any other storage device.

[0046] The processor 402 performs processing of each device by reading and executing programs from the memory 401. The processor 402 may be, for example, a microprocessor, a microprocessor unit (MPU), or a central processing unit (CPU). The processor 402 may include multiple processors.

[0047] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0048] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0049] For example, the combinations of explanatory variables and response variables in the kernel ridge regression model are not limited to those described in the above embodiment. For example, the explanatory variables may be information obtained by performing segmentation processing on an RGB image. Furthermore, the response variable may be the position and orientation of the pallet 200. In this case, the position and orientation estimation unit 125 shown in FIG. 4 may be omitted. Furthermore, the target object is not limited to the pallet 200, a so-called flat pallet, as described above, but may be anything that can be handled by a forklift. For example, the target object may be a box pallet, a roll box pallet, a post pallet, or any other shape of pallet. Furthermore, an RGB-D camera may be used instead of the RGB cameras 11 and 121.

[0050] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0051] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A position and orientation estimation device comprising: an image acquisition unit mounted on a forklift and configured to acquire an RGB image; a front surface information estimation unit configured to estimate front surface information, which is information related to the front surface of an object, from the RGB image; an inference unit configured to infer the position and orientation related information from the front surface information estimated by the front surface information estimation unit using a trained regression model using the front surface information and position and orientation related information, which is information related to the position and orientation of the object, as training data, and to calculate a reliability of the inferred position and orientation related information; and a removal unit configured to remove the position and orientation related information as an outlier if the reliability calculated by the inference unit is equal to or less than a predetermined threshold. (Supplementary Note 2) The position and orientation estimation device according to Supplementary Note 1, further comprising: an image acquisition unit configured to acquire an RGB image; a front surface information estimation unit configured to estimate the position and orientation of the object based on the front surface information and the position and orientation related information inferred by the inference unit. (Supplementary Note 3) The position and orientation estimation device according to Supplementary Note 1, wherein the front surface information estimation unit estimates the front surface information using YOLO (You Only Look Once). (Supplementary Note 4) The position and orientation estimation device according to Supplementary Note 1, wherein the regression model is a kernel ridge regression model. (Supplementary Note 5) A position and orientation estimation method, comprising: a computer estimating front surface information, which is information relating to a front surface of an object, from an RGB image captured by an RGB camera mounted on a forklift; inferring the position and orientation related information from the estimated front surface information using a regression model that has been trained using the front surface information and position and orientation related information, which is information relating to the position and orientation of the object, as training data; and calculating reliability of the inferred position and orientation related information; and removing the position and orientation related information as an outlier if the calculated reliability is equal to or less than a predetermined threshold.(Supplementary Note 6) A position and orientation estimation program that causes a computer to execute the following processes: estimating front surface information, which is information about the front surface of an object, from an RGB image captured by an RGB camera mounted on a forklift; inferring position and orientation related information from the estimated front surface information using a trained regression model using the front surface information and position and orientation related information, which is information about the position and orientation of the object, as training data, and calculating reliability of the inferred position and orientation related information; and removing the position and orientation related information as an outlier if the calculated reliability is equal to or less than a predetermined threshold. Some or all of the elements (e.g., configurations and functions) described in Supplements 2 to 4 that are dependent on Supplementary Note 1 may also be dependent on Supplements 5 and 6 in the same dependent relationship as Supplements 2 to 4. Some or all of the elements described in any of the Supplements may be applied to various hardware, software, and recording means, systems, and methods for recording software.

[0052] This application claims priority based on Japanese Patent Application No. 2024-001645, filed January 10, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0053] It is possible to provide a position and orientation estimation device, a position and orientation estimation method, and a position and orientation estimation program that can estimate the position and orientation of an object more inexpensively and with high accuracy.

[0054] 10, 100 Position and orientation estimation device 110 Pre-processing unit 111 Camera position and orientation calculation unit 112 Pallet position and orientation calculation unit 113 Teacher data generation unit 114 Learning unit 120 Real-time processing unit 11, 121 RGB camera (image acquisition unit) 12, 122 Front surface information estimation unit 13, 123 Inference unit 14, 124 Removal unit 125 Position and orientation estimation unit 130 Model storage unit 200 Pallet (object) 301, 302, 303, 304 Baggage P1 to P11 Corner points B1 to B3 Bounding box A1 to A4 Area

Claims

1. An apparatus for estimating a position and orientation, comprising: an image acquisition means mounted on a forklift for acquiring an RGB image; a front surface information estimation means for estimating front surface information which is information about the front surface of an object from the RGB image; an inference means for inferring the position and orientation related information which is information about the position and orientation of the object from the front surface information estimated by the front surface information estimation means by using a regression model learned with the front surface information and the position and orientation related information as teacher data, and calculating a reliability of the inferred position and orientation related information; and a removal means for removing the position and orientation related information as an outlier when the reliability calculated by the inference means is equal to or less than a predetermined threshold value.

2. The apparatus for estimating a position and orientation according to claim 1, further comprising a position and orientation estimation means for estimating the position and orientation of the object based on the front surface information and the position and orientation related information inferred by the inference means, wherein the position and orientation related information is information about the rear surface of the object.

3. The apparatus for estimating a position and orientation according to claim 1, wherein the front surface information estimation means estimates the front surface information by using YOLO (You Only Look Once).

4. The apparatus for estimating a position and orientation according to claim 1, wherein the regression model is a kernel ridge regression model.

5. A method for estimating a position and orientation, comprising: estimating front surface information which is information about the front surface of an object from an RGB image captured by an RGB camera mounted on a forklift; inferring the position and orientation related information which is information about the position and orientation of the object from the estimated front surface information by using a regression model learned with the front surface information and the position and orientation related information as teacher data, and calculating a reliability of the inferred position and orientation related information; and removing the position and orientation related information as an outlier when the calculated reliability is equal to or less than a predetermined threshold value.

6. The method for estimating a position and orientation according to claim 5, wherein the position and orientation related information is information about the rear surface of the object, and the computer estimates the position and orientation of the object based on the front surface information and the inferred position and orientation related information.

7. The method for estimating a position and orientation according to claim 5, wherein the computer estimates the front surface information by using YOLO (You Only Look Once).

8. The position and orientation estimation method according to claim 5, wherein the regression model is a kernel ridge regression model.

9. A position and orientation estimation program for causing a computer to execute: a process of estimating front surface information, which is information about the front surface of an object, from an RGB image captured by an RGB camera mounted on a forklift; a process of inferring the position and orientation related information, which is information about the position and orientation of the object, from the estimated front surface information using a regression model that has been learned using the front surface information and the position and orientation related information as teacher data, and calculating the reliability of the inferred position and orientation related information; and a process of removing the position and orientation related information as an outlier when the calculated reliability is equal to or less than a predetermined threshold.

10. The position and orientation estimation program according to claim 9, wherein the position and orientation related information is information about the rear surface of the object, and causing the computer to execute a process of estimating the position and orientation of the object based on the front surface information and the inferred position and orientation related information.

11. The position and orientation estimation program according to claim 9, wherein in the process of estimating the front surface information, the computer estimates the front surface information using YOLO (You Only Look Once).

12. The position and orientation estimation program according to claim 9, wherein the regression model is a kernel ridge regression model.

Images