Article location registration device, article location registration system, and article search device

Abstract

To provide an article location registration device that accurately calculates a location of a real article even in a place where GPS information cannot be obtained.SOLUTION: The article location registration device for calculating a location of an article for an information code, includes: a first calculation unit that calculates a first location and a first orientation of the article location registration device represented in reference coordinates with the origin at the information code on the basis of an image of the information code and a distance between an image acquiring unit and the information code; and a second calculation unit that calculates a second location representing a current location of the article location registration device represented in the reference coordinates and a second orientation representing a current orientation on the basis of the first location and the first orientation calculated by the first calculation unit, location information detected by a first sensor and angular information detected by a second sensor, and calculates the location of the article represented in the reference coordinates on the basis of an image representing the vicinity of the article acquired by the image acquiring unit, a distance between the image acquiring unit and the vicinity of the article, and a second location and a second orientation.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an item location registration device, an item location registration system, and an item search device. [Background technology]

[0002] 2. Description of the Related Art Conventionally, augmented reality technology is used to place an augmented reality object on an image that represents a real object on the ground.

[0003] For example, it has been proposed to use an augmented reality technology to provide route guidance to reach a destination (see, for example, Patent Document 1). In Patent Document 1, the position of a real object on the ground is expressed in world coordinates (also called global coordinates) using GPS (Global Positioning System) information. The world coordinates have their origin at a predetermined position on the ground. The GPS information is acquired by receiving radio waves from satellites.

[0004] Here, an augmented reality object is displayed on the image of the terminal in correspondence with a real object on the ground. The position of the real object is calculated based on the position and orientation of the terminal estimated using GPS information, but an error may occur.

[0005] Patent Document 1 proposes using AR anchors placed at predetermined positions on the ground together with GPS information to correct the position of a terminal and calculate the correct position of a real-world object. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2020 / 227734 Summary of the Invention [Problem to be solved by the invention]

[0007] Meanwhile, it is desirable to use augmented reality technology even in areas where GPS information cannot be acquired or received, such as indoors or underground.

[0008] In order to use augmented reality technology, it is preferable to accurately calculate the positions of real objects even indoors where GPS information cannot be obtained.

[0009] Therefore, an object of the present disclosure is to provide an article position registration device that can correctly calculate the actual position of an article even in a place where GPS information cannot be obtained. [Means for solving the problem]

[0010] According to one embodiment, an article position registration device is provided, which is an article position registration device that calculates the position of an article relative to an information code, and is characterized by having an image acquisition unit, a first sensor that detects position information for calculating a position change amount, a second sensor that detects angle information for calculating an angle change amount, a first calculation unit that calculates a first position and a first orientation of the article position registration device expressed in reference coordinates having an origin at the information code based on an image of the information code acquired by the image acquisition unit and a distance between the image acquisition unit and the information code, and a second calculation unit that calculates a second position representing a current position of the position calculation device expressed in reference coordinates and a second orientation representing a current orientation based on the first position and first orientation calculated by the first calculation unit, position information detected by the first sensor, and angle information detected by the second sensor, and calculates the position of the article expressed in reference coordinates based on an image representing the vicinity of the article acquired by the image acquisition unit, a distance between the image acquisition unit and the vicinity of the article, and the second position and second orientation.

[0011] According to another embodiment, an item management system is provided, which is an item position registration device that calculates the position of an item relative to a reference code, and includes an image acquisition unit, a first sensor that detects position information for calculating a position change amount, a second sensor that detects angle information for calculating an angle change amount, a first calculation unit that calculates a first position and a first orientation of the item position registration device expressed in reference coordinates having an origin on the information code based on an image of the information code acquired by the image acquisition unit and a distance between the image acquisition unit and the information code, and a first position and a first orientation calculated by the first calculation unit and the detected angle by the first sensor. The article position registration device has a second calculation unit that calculates a second position representing the current position of the article position registration device expressed in a reference coordinate system and a second orientation representing the current orientation based on position information and angle information detected by a second sensor, and calculates the position of the article expressed in the reference coordinates based on an image representing the vicinity of the article acquired by the image acquisition unit, the distance between the image acquisition unit and the vicinity of the article, and the second position and the second orientation, and a memory unit that stores the position of the article expressed in the reference coordinate system calculated by the article position registration device.

[0012] According to another embodiment, an item search device is provided, which searches for an item whose position is calculated by the above-mentioned item position registration device, and is characterized by having a display unit, a second image acquisition unit, a second first sensor that detects position information for calculating a position change amount, a second second sensor that detects angle information for calculating an angle change amount, a determination unit that determines whether or not the position of the item represented by the reference coordinates is included in the field of view of the second image acquisition unit, a generation unit that generates an expanded image in which item information related to the item is superimposed on the image acquired by the second image acquisition unit when the determination unit determines that the position of the item represented by the reference coordinates is included in the field of view of the second image acquisition unit, and a control unit that displays the expanded image generated by the generation unit on the display unit. Effect of the Invention

[0013] The article position registration device according to the present invention has the advantage that the actual position of an article can be correctly calculated and registered even in a place where GPS information cannot be obtained. [Brief description of the drawings]

[0014] [Figure 1] 1 is a schematic configuration diagram of an item management system in which a terminal including an item management unit is implemented. [Diagram 2] FIG. 11 is a sequence diagram of an item location registration process in the item management system. [Diagram 3] FIG. 2 is a hardware configuration diagram of a terminal that registers the position of an item. [Figure 4] FIG. 1A is a diagram showing a code management table, and FIG. 1B is a diagram showing an article management table. [Diagram 5] FIG. 11 is a diagram illustrating a process for calculating the position of a terminal. [Figure 6] FIG. 4A is a diagram for explaining an example of a process for registering the position of an item, and FIG. 4B is a diagram for explaining another example of a process for registering the position of an item. [Figure 7] FIG. 2 is a hardware configuration diagram of a terminal that searches for an item. [Figure 8] FIG. 11 is a sequence diagram of an item search process in the item management system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Fig. 1 is a schematic configuration diagram of an item management system 1 having a terminal 10 in which an item management unit 17 is implemented. Fig. 2 is a sequence diagram of an item position registration process of the item management system 1. Fig. 3 is a hardware configuration diagram of the terminal 10.

[0016] The item management system 1 includes at least one terminal 10 and a server 30. An item management unit 17 is implemented in the terminal 10. The terminal 10 is connected to the server 30 via the macrocell base station 41 and the communication network 40 by, for example, accessing a wireless base station 41 (hereinafter also referred to as a macrocell base station 41) that provides a macrocell that is connected to the communication network 40 to which the server 30 is connected via a gateway (not shown) or the like. The terminal 10 is an example of an item position registration device.

[0017] 1, only one terminal 10 is shown, but the article management system 1 may include multiple terminals 10. Similarly, multiple macrocell base stations 41 may be connected to the communication network 40.

[0018] A user (not shown) of the item management system 1 manages the positions of multiple items 60 placed in a warehouse 50 using a terminal 10 that registers the positions of the items. For example, the user uses the terminal 10 to register the positions of the items 60 in the warehouse 50 in the server 30. It is not always possible to receive GPS information inside the warehouse 50. Therefore, the item management system 1 manages the positions of the items 60 without using GPS information, as described below.

[0019] Next, the item location registration process of the item management system 1 will be described below with reference to Fig. 2. First, the user uses the camera 13 of the terminal 10 to capture an image of the reference code C1 displayed near the entrance of the warehouse 50 (step S101). The vicinity here refers to an area in which the item, the reference code C1, the item code D1, the item 60, etc. can be identified. The reference code C1 is an example of an information code.

[0020] The reference code C1 represents the origin of a reference coordinate S1 that represents the position of the item 60 within the warehouse 50. It is preferable to select the entrance / exit of the warehouse 50 as the position where the reference code C1 is displayed, from the viewpoint that a user who enters the warehouse 50 can obtain an image of the reference code C1.

[0021] The reference code C1 may be displayed on a medium such as paper, etc. Also, the reference code C1 may be displayed on a display device.

[0022] The reference code C1 may be a code having information such as a one-dimensional code or a two-dimensional code. For example, the code described in Japanese Patent No. 5908113 may be used as the reference code C1.

[0023] Next, the terminal 10 analyzes the image of the reference code C1 to obtain the code ID contained in the code, and transmits this code ID to the server 30 via the macrocell base station 41 and the communication network 40 (step S102).

[0024] Next, the server 30 receives the code ID and transmits the dimensions of the reference code C1 associated with this code ID to the terminal 10 via the communication network 40 and the macrocell base station 41 (step S103). For example, when the reference code C1 is a rectangular two-dimensional code, the dimensions of the reference code C1 represent the vertical and horizontal lengths.

[0025] Next, the terminal 10 calculates the position and orientation of the terminal 10 expressed in reference coordinates S1 having the origin at the reference code C1 based on the image showing the reference code C1 and the distance between the camera 13 of the terminal 10 and the reference code C1 (step S104). The process of calculating the position of the terminal 10 will be described in detail later.

[0026] Next, with the camera 13 activated, the user moves together with the terminal 10 to the location of the item 60 whose location is to be registered (step S105).

[0027] Next, the user acquires an image showing the vicinity of the item 60 using the camera 13 of the terminal 10 (step S106). The image showing the vicinity of the item 60 is an image showing the item or an image showing an item code. The vicinity here refers to a range in which the user can visually recognize the item as the target item, including the position of the item itself. The terminal 10 has a first sensor 14 that detects position information for determining the amount of change in position of the terminal 10, and a second sensor 15 that detects angle information for determining the amount of change in angle of the terminal 10. The amount of movement of the terminal 10 from the position where the image of the reference code C1 was acquired is obtained by accumulating the position information detected by the first sensor 14. The amount of change in orientation of the terminal 10 from the orientation when the image of the reference code C1 was acquired is obtained by accumulating the angle information detected by the second sensor 15.

[0028] Next, the terminal 10 calculates the current position and orientation of the terminal 10 based on the position and orientation when the image of the reference code C1 was acquired, the position information detected by the first sensor 14, and the angle information detected by the second sensor 15. The terminal 10 also calculates the position of the item 60 represented by the reference coordinates S1 based on the current position and orientation of the terminal 10, the image of the vicinity of the item 60 acquired by the camera 13, and the distance between the camera 13 and the vicinity of the item 60 (step S107). Details of these processes of the terminal 10 will be described later.

[0029] Next, the terminal 10 transmits the position of the item 60 expressed in the reference coordinates S1 together with the item ID to the server 30 via the macrocell base station 41 and the communication network 40 (step S108).

[0030] Next, the server 30 registers the position of the item 60 represented by the reference coordinates S1 in the item management table 200 (see FIG. 4(B)), and ends the series of processes.

[0031] According to the item management system 1 of this embodiment described above, the position of the item 60 is represented by a reference coordinate S1 with the reference code C1 displayed in the warehouse 50 as the origin, so that the actual position of the item can be correctly calculated even in a warehouse 50 where GPS information cannot be obtained.

[0032] The article management system 1 may include multiple terminals 10, but since each terminal has the same configuration and executes the same processes with respect to article management processing, the following description will focus on one terminal 10.

[0033] Next, the terminal 10 on which the terminal 10 is implemented will be described below with reference to Fig. 3. Fig. 3 is a hardware configuration diagram of the terminal 10 that registers the position of the item 60. The terminal 10 has a communication unit 11, a user interface (IF) 12, a camera 13, a first sensor 14, a second sensor 15, a third sensor 16, an item management unit 17, etc.

[0034] The communication unit 11, the user IF 12, the camera 13, the first sensor 14, the second sensor 15, the third sensor 16, and the item management unit 17 are communicatively connected via a signal line 18.

[0035] The communication unit 11 is an example of a terminal communication unit, and has an interface circuit for connecting the terminal 10 to the macrocell base station 41. The communication unit 11 is configured to be able to communicate with the server 30 via the macrocell base station 41 and the communication network 40.

[0036] The user IF 12 displays processing information associated with processing of the terminal 10, and inputs operation information associated with user operations. The processing information includes, for example, an image captured by the camera 13. The user IF 12 includes, for example, a touch panel.

[0037] Camera 13 is an example of an image acquisition unit. Camera 13 is attached to terminal 10 so as to face outward from terminal 10. Camera 13 outputs the acquired image to item management unit 17 via signal line 18. Camera 13 has a two-dimensional detector configured with an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector.

[0038] The first sensor 14 detects position information for determining the amount of change in position of the terminal 10, and outputs this position information to the item management unit 17 via a signal line 18. As the first sensor 14, for example, an acceleration sensor can be used.

[0039] The second sensor 15 detects angle information for determining the amount of change in angle of the terminal 10, and outputs this angle information to the item management unit 17 via a signal line 18. As the second sensor 15, for example, a gyroscope can be used.

[0040] The third sensor 16 is an example of a distance measuring unit. The third sensor 16 detects the distance between the item 60 and the terminal 10, and outputs the distance to the item management unit 17 via the signal line 18. As the third sensor 16, for example, a LiDAR sensor can be used.

[0041] The article management unit 17 executes control processing and calculation processing. To this end, the terminal 10 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the terminal 10 to the signal line 18.

[0042] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores computer programs of applications used in the information processing executed by the processor 23 and various data.

[0043] All or part of the functions of the terminal 10 are functional modules realized by, for example, a computer program running on the processor 23. The processor 23 has a control unit 231 and a calculation unit 232. The calculation unit 232 is an example of a first calculation unit, a second calculation unit, and a third calculation unit. The functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 has one or more central processing units (CPUs) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit. The terminal 10 is, for example, an electronic control unit (ECU). Details of the operation of the terminal 10 will be described later.

[0044] The server 30 executes a control process, an update process, and a search process. To that end, the server 30 has a communication interface (IF) 31, a storage device 32, a memory 33, and a processor 34. The communication IF 31, the storage device 32, and the memory 33 are connected to the processor 34 via a signal line 35. The server 30 may further have input devices such as a keyboard and a mouse, and a display device such as a liquid crystal display.

[0045] The communication IF 31 has an interface circuit for connecting the server 30 to the communication network 40. The communication IF 31 is configured to be able to communicate with the terminal 10 via the communication network 40 and the macrocell base station 41. The communication IF 31 is an example of a server communication unit.

[0046] The storage device 32 includes, for example, a hard disk device or an optical recording medium and an access device therefor. The storage device 32 stores a code management table 100 (see FIG. 4(A)) for managing the reference code C1 and the like, and an item management table 200 (see FIG. 4(B)). The storage device 32 may further store identification information for identifying the terminal 10. Furthermore, the storage device 32 may store a computer program for executing the processing of the server 30 related to the item management processing, which is executed on the processor 34. The storage device 32 is an example of a memory unit.

[0047] 4A is a diagram showing a code management table 100. The code management table 100 has a management number column 101, a code ID column 102, a type column 103, a position column 104, a transformation matrix column 105, and a specification column 106.

[0048] In the management number field 101, a management number for managing a code managed by the article management system 1 is registered. In the code ID field 102, a code ID for identifying the code is registered. In the type field 103, a type of the code is registered. There are three types of codes: an origin code representing the origin of the reference coordinate S1, such as the reference code C1; a fixed code representing the origin of a local auxiliary coordinate fixed in the warehouse; and an indefinite code representing the origin of a local auxiliary coordinate movable in the warehouse. The fixed code and the indefinite code will be described later. In the position field 104, a position of the code represented by the reference coordinate S1 is registered. In the transformation matrix field 105, a transformation matrix for transforming a position represented by the auxiliary coordinate to a position represented by the reference coordinate S1 is registered. In the specification field 106, the dimensions, version, error correction rate, etc. of the code are registered.

[0049] 4(B) is a diagram showing an item management table 200. In the item management table 200, information regarding the location of the item 60 placed in the warehouse 50 is registered. The item management table 200 has a management number column 201, an item ID column 202, a registration information column 203, a location column 204, a specification column 205, and a status column 206. The item management table 200 may also register the name of the item.

[0050] The management number column 101 registers a management number for managing an item managed by the item management system 1. The item ID column 202 registers an item ID for identifying an item. The registration information column 203 registers registration information indicating a method for registering an item's position; the registration information will be described later. The position column 204 registers information indicating the position of the item expressed in the reference coordinates S1. In the specification column 205, when the position of the item is represented using an item code, the dimensions, version, error correction rate, etc. of the item code are registered. The status column 206 registers whether the item is in a found state or an unsearched state.

[0051] The memory 33 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 33 stores computer programs of applications and various data used in information processing executed by the processor 34. Note that the restriction information may be stored in the memory 33. The memory 33 is an example of a storage unit.

[0052] All or part of the functions of the server 30 are functional modules realized by, for example, a computer program running on the processor 34. The processor 34 has a control unit 341, a registration unit 342, and a search unit 343. Alternatively, the functional modules of the processor 34 may be dedicated arithmetic circuits provided in the processor 34. The processor 34 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 34 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit.

[0053] When the control unit 341 receives a code ID from the terminal 10, it refers to the code management table 100 stored in the storage device 32, reads out the dimensions associated with this code ID, and transmits them to the terminal 10.

[0054] When the registration unit 342 receives the position and the item ID of the item 60 from the terminal 10, it associates the position of the item 60 with the item ID of the item 60 and registers it in the item management table 200 stored in the storage device 32. Details of the operation of the search unit 343 will be described later.

[0055] Next, the process of calculating the position of the terminal 10 in the above-mentioned step S104 will be described below. The calculation unit 232 of the terminal 10 calculates the first position P1 and the first orientation A1 of the terminal 10 expressed in the reference coordinates S1 having the origin at the reference code C1, based on the image showing the reference code C1 and the distance between the camera 13 of the terminal 10 and the reference code C1.

[0056] The calculation unit 232 uses the third sensor 16 to calculate the distance between the third sensor 16 and the reference code C1. The positional relationship between the third sensor 16 and the camera 13 is stored as an internal parameter in the memory 22. The calculation unit 232 calculates the distance between the camera 13 and the reference code C1 based on the distance between the third sensor 16 and the reference code C1 and the internal parameter.

[0057] Furthermore, the calculation unit 232 receives the dimensions of the reference code C1 from the server 30. The reference code C1 has a square outer shape. The vertical dimension and the horizontal dimension of the reference code C1 are the same. If the reference code C1 includes the dimensions of the reference code C1 as information, the calculation unit 232 may obtain the dimensions of the reference code C1 by analyzing an image of the reference code C1.

[0058] The calculation unit 232 may also calculate the distance between the camera 13 and the reference code C1 based on the dimensions of the reference code C1 and the number of pixels expressing the reference code C1 shown in the image.

[0059] 5 is a diagram for explaining the process of calculating the position of the terminal 10. The camera coordinate S2 having an origin O2 on the lens of the camera 13 has a Z axis in the direction of the optical axis of the lens of the camera 13, and has an X axis and a Y axis that are orthogonal to each other at the origin O2 of this Z axis.

[0060] The reference code C1 has position detection patterns at the four corners of a square. In the camera coordinate system S2, the center points of the four position detection patterns of the reference code C1 are represented as A (X1, Y1, Z1), B (X2, Y2, Z2), C (X3, Y3, Z3), and D (X4, Y4, Z4).

[0061] The image coordinate S3 is placed in a plane located at the focal length from the origin on the Z axis of the camera coordinate S2. The origin O3 of the image coordinate S3 is placed in the upper left corner. The image coordinate S3 is set within a range of w in the width direction from the origin O3 and h in the downward direction from the origin O3. The positions of the four corners of the image coordinate S3 are (0,0), (w,0), (w,h), and (w,0). The intersection of the optical axis and the plane containing the image coordinate S3 is represented by (w / 2, h / 2) in the image coordinate S3.

[0062] In the image coordinate S3, the image of the reference code C1 is the area surrounded by the intersection of a straight line connecting the origin O2 of the camera coordinate S2 and the center points A to D of the four position detection patterns of the reference code C1 with a plane including the image coordinate S3.

[0063] In the image coordinate S3, the central points of the four position detection patterns of the image of the reference code C1 are represented by a(x1, y1), b(x2, y2), c(x3, y3), and d(x4, y4).

[0064] In the camera coordinate system S2, vector Sa pointing from the origin O2 to point a (x1, y1), vector Sb pointing from the origin O2 to point b (x2, y2), vector Sc pointing from the origin O2 to point c (x3, y3), and vector Sd pointing from the origin O2 to point d (x4, y4) are expressed by the following formula 1. Here, z represents the distance from the origin O2 to the focal point of the lens.

[0065]

number

[0066] Here, each of the central points of the four position detection patterns of reference code C1 is located on an extension of a straight line connecting the central points of the four position detection patterns of the image of reference code C1 to origin O2. The vectors pointing from origin O2 to central points A to D of the four position detection patterns of reference code C1 are expressed by the following formula 2.

[0067]

number

[0068] Here, k, l, m, and n are real numbers equal to or greater than 1. In addition, since the reference code C1 is a square, the following formula 3 holds.

[0069]

number

[0070] Here, if the horizontal dimension of the reference code C1 is W and the vertical dimension is H, the following formula 4 holds.

[0071]

number

[0072] The calculation unit 232 solves this simultaneous equation to obtain the positions of the center points A, B, C, and D of the four position detection patterns of the reference code C1 expressed in the camera coordinate system S2.

[0073] Next, the calculation unit 232 transforms the coordinates expressed in the camera coordinates S2 into the reference coordinates S1. An affine transformation including translation and rotation of three-dimensional coordinates can be expressed by a matrix product.

[0074] Here, if the reference code C1 represented in the camera coordinate S2 is the origin (tx, ty, tz), the vector pointing along the X axis of the reference coordinate S1 represented in the camera coordinate S2 is (Xx, Xy, Xz), the vector pointing along the Y axis of the reference coordinate S1 is (Yx, Yy, Yz), and the vector pointing along the Z axis of the reference coordinate S1 is (Zx, Zy, Zz), then the matrix M1 that converts the coordinates represented in the camera coordinate S2 to the reference coordinate S1 is expressed by the following equation 5.

[0075]

number

[0076] With the reference code C1 as the origin, for example, the position of the center of the square can be used. The direction from the midpoint of the line segment CD to the midpoint of the line segment AB can be used as the Y-axis vector of the reference coordinate S1. The direction of the cross product of vectors AB and AD can be used as the Z-axis vector of the reference coordinate S1. Note that the origin of the reference code C1 may be a position other than the center of the square.

[0077] By using the above formula (5), the position of the origin O2 expressed in the camera coordinate S2 is converted to the position of the reference coordinate S1, so that the position of the lens of the camera 13 can be expressed in the reference coordinate S1. In this embodiment, the position of the terminal 10 is expressed by the position of the lens of the camera 13. In this way, the calculation unit 232 calculates the position of the terminal 10 expressed in the reference coordinate S1. Note that the position of the terminal 10 may be expressed by applying a predetermined offset to the position of the lens of the camera 13.

[0078] When the position of the terminal 10 represented by the reference coordinates S1 is calculated, the control unit 231 may display on the user IF 12 that the position of the terminal 10 has been calculated, to notify the user.

[0079] Next, the process of calculating the position of the article 60 in step S107 described above will be described below.

[0080] After acquiring the image of the reference code C1, the user moves to the article 60 whose position is to be registered. The terminal 10 also moves together with the user. The calculation unit 232 obtains the amount of movement from the position of the terminal 10 when the position of the terminal 10 in the reference coordinates S1 is calculated to the article 60 by integrating the position information detected by the first sensor 14. The calculation unit 232 also obtains the amount of change in orientation from the position of the terminal 10 when the position of the terminal 10 in the reference coordinates S1 is calculated to the position where the image of the article 60 is acquired by integrating the angle information detected by the second sensor 15.

[0081] Such a technology for estimating the self-position using the amount of movement and the amount of change in orientation of the terminal 10 is known as PDR (Pedestrian Dead Reckoning). It is also realized as an IMU (Inertial Measurement Unit) combining sensors such as an acceleration sensor and a gyro sensor. Here, it is preferable from the viewpoint of obtaining accurate values ​​of the amount of movement and the amount of change in orientation of the terminal 10 to correct the amount of movement and the amount of change in orientation using an image acquired by the camera 13 while the terminal 10 is moving. Such a technology is known as VIO (Visual-Inertial Odometry). Software or modules using VIO include AR frameworks, such as ARkit (registered trademark). The AR framework has various functions for developing an AR application. Here, it is assumed that the AR framework has a function of displaying an image captured by a camera on the screen of the terminal by perspective transformation and a tracking (movement distance detection) function using various sensors (LiDAR, gyroscope, linear accelerometer, etc.) and a camera.

[0082] Here, a matrix Mw that converts a position expressed by coordinates SA (see FIG. 1) with the position of camera 13 at the time of starting camera 13 as the origin into a position of reference coordinates S1 is expressed by the following formula 6.

[0083]

number

[0084] Matrix M2 converts a position expressed by coordinate SA (see FIG. 1) with the position of camera 13 at the time of starting camera 13 as the origin, into the position of camera coordinate S2 of camera 13 of terminal 10 located at any position within warehouse 50. Coordinate SA is set with the camera position of camera 13 at the time of starting camera 13 as the origin, and the camera direction as the azimuth. The any position within warehouse 50 is the position of camera 13 when the image of reference code C1 was captured.

[0085] Usually, the camera 13 is activated before acquiring an image of the reference code C1. The matrix M2 is calculated based on the position of the terminal 10 and the orientation of the camera 13 at the time when the camera 13 is activated as the terminal 10 moves, and the amount of movement and orientation change of the terminal 10 thereafter. The matrix M2 can be calculated in the same manner as in the above formula 5. For example, an AR framework, specifically ARkit (registered trademark), etc. can be used to calculate the matrix M2.

[0086] The user uses the camera 13 of the terminal 10 to capture an image showing the vicinity of the article 60. On the screen of the user IF 12 on which the image of the article 60 is displayed, the user taps a point in the area in which the article 60 is displayed. The image showing the vicinity of the article 60 preferably includes the article 60.

[0087] The calculation unit 232 detects the distance to the article 60 in the direction of the tapped position in the image coordinates S3 using the third sensor 16. The distance between the tapped position in the image coordinates S3 and the camera 13 is an example of the distance between the camera 13 and the vicinity of the article.

[0088] A method in which the calculation unit 232 calculates the position of the article 60 based on the tapped position will be described below. In the camera coordinates S2, a vector Sa connecting the origin O2 and the tapped position on the image coordinates S3 is expressed, for example, by the following formula 7. Here, it is assumed that the user taps a certain point a on the image of the article 60.

[0089]

number

[0090] Since the article 60 is on the extension of the vector Sa that points from the origin O2 to the point a(x1, y1) in the camera coordinate system S2, the following formula 8 holds: The vector SA points from the origin O2 to a point on the article 60 that corresponds to the tapped point.

[0091]

number

[0092] Here, k is a real number equal to or greater than 1. If the distance to the article 60 in the direction of the tapped position in the image coordinate S3 is L, then the following formula 9 holds. L in the following formula 9 is detected using the third sensor 16 (distance measuring unit).

[0093]

number

[0094] The calculation unit 232 solves this simultaneous equation to determine the position of a point C1 on the article 60 expressed in the camera coordinates S2.

[0095] Then, the calculation unit 232 converts the position C1 expressed in the camera coordinates S2 into a position in the reference coordinates S1.

[0096] Matrix Mcw that converts position C1 expressed in camera coordinates S2 into a position in reference coordinates S1 is expressed by the following formula 10. Here, matrix M2 converts a position expressed in coordinates SA, which has the origin at the position of camera 13 when camera 13 is started, into a position in camera coordinates S2 of camera 13 of terminal 10 located at an arbitrary position in warehouse 50. The arbitrary position in warehouse 50 is a position where an image of item 60 is captured by camera 13.

[0097]

number

[0098] The calculation unit 232 uses Mcw expressed by the above formula 10 to convert the position C1 expressed in the camera coordinates S2 into the position B1 in the reference coordinates S1, as expressed by the following formula 11.

[0099]

number

[0100] The control unit 231 transmits the position B1 of the item 60 together with the item ID and the registration information for identifying the item 60 to the server 30. The item ID and the registration information are input or selected from a predetermined list by the user operating the user IF 12.

[0101] The registration information registered in the registration information column 203 of the item management table 200 is a one-point tap because one point in the area showing the item 60 has been tapped. In the one-point tap, as described above, the calculation unit 232 calculates the position of the item shown in the reference coordinates S1 based on the one point selected on the image of the item 60 shown on the screen of the user IF 12.

[0102] This allows the user to select the target item 60 without placing a burden on the user.

[0103] It should be noted that the method of registering the location of the article may be other than the above. Next, the other registration method will be described below.

[0104] (4-point tap) The calculation unit 232 may calculate, as the position of the article 60, a plane represented by reference coordinates S1 defined based on four points selected on an image representing the vicinity of the article 60 displayed on the screen of the user IF 12. The distance between the camera 13 and the four points selected on the image representing the vicinity of the article 60 is an example of the distance between the camera 13 and the vicinity of the article.

[0105] This reduces the burden on the user while enabling the acquisition of object posture information that cannot be calculated with a single tap.

[0106] On the screen of the user IF 12 on which an image of the article 60 is displayed, the user taps four points in the area in which the article 60 is displayed.

[0107] The calculation unit 232 uses the third sensor 16 to detect the distance between the camera 13 and the article 60 located in the direction of the four tapped points in the image coordinate S3.

[0108] The calculation unit 232 obtains four positions B1 to B4 of the reference coordinate S1 based on the four tapped points on the image in the same manner as described above. The plane represented by the reference coordinate S1 is defined by the four positions B1 to B4. The position of the article 60 is represented by this plane.

[0109] The control unit 231 transmits the positions B1 to B4 of the item 60 together with the item ID for identifying the item 60 and the registration information to the server 30. The registration information is four-point tap.

[0110] The calculation unit 232 may calculate, as the position of the article 60, a plane represented by a reference coordinate S1 defined based on at least three points selected on the image of the article 60 displayed on the screen of the user interface 12.

[0111] This makes it possible to obtain a planar image of an item with a minimum of three points, and by using the number of taps to obtain the outline of the item, the image can be cut out and associated with the item and registered in the item management table 200 in Figure 4(B), thus improving the convenience of management.

[0112] The calculation unit 232 may generate a distance image using the third sensor 16 for the area on the image coordinates S3 where the item 60 is displayed, select an area where the distance from the terminal 10 is within a predetermined reference value range, and set the plane representing this area as the position of the item 60. In this case, the calculation unit 232 may also select four points on the plane representing this area, and transmit the positions of these four points to the server 30 as the position of the item 60.

[0113] (First Code) 6(A) is a diagram for explaining an example of a process for registering the position of an item 60. An item code D1 is placed on the item 60. The item code D1 is an example of a first item code.

[0114] The calculation unit 232 calculates a plane containing the item code D1 represented by the reference coordinates S1 based on the image of the item code D1 shown in the image acquired by the camera 13, the distance between the camera 13 and the item code D1, and the current position and current orientation of the terminal 10 calculated by the calculation unit 232, and determines that a specified area within the plane containing the item code D1 is the position of the item 60 represented by the reference coordinates S1.

[0115] The camera 13 of the terminal 10 is operated by a user to capture an image showing the vicinity of the item 60 including the item code D1. The item code D1 is a two-dimensional code having a rectangular outer shape. Position detection patterns are arranged at the four corners of the item code D1.

[0116] The calculation unit 232 analyzes the image of the item code D1 to obtain information contained in the code, and based on this information, determines that the item code D1 is a code for calculating the position of the item 60. The calculation unit 232 detects the distance between the camera 13 and the item code D1 in the direction of the center points of the four position detection patterns of the item code D1 in the image coordinate S3, using the third sensor 16.

[0117] The calculation unit 232 calculates the position of the center point of each of the four position detection patterns of the item code D1 represented by the camera coordinate S2, and then calculates the position of the center point of each of the four position detection patterns of the item code D1 represented by the reference coordinate S1. The positions of the center points of each of the four position detection patterns of the item code D1 define a plane including the item code D1. The calculation unit 232 determines a predetermined area in the plane represented by the center points of each of the four position detection patterns of the item code D1 as the position of the item 60 represented by the reference coordinate S1. The predetermined area in the plane may be, for example, an area surrounded by the center points of each of the four position detection patterns of the item code D1.

[0118] The control unit 231 transmits the positions of the center points of each of the four position detection patterns of the item code D1 to the server 30 together with the item ID for identifying the item 60, the registration information, and the specifications of the item code D1. The registration information is a first code. The specifications of the item code D1 are input or selected from a predetermined list by the user operating the user IF 12.

[0119] As a result, when a two-dimensional code is used to register the position of the article 60, highly accurate position information and orientation can be calculated without the user having to perform an action such as tapping.

[0120] (Second Code) 6(B) is a diagram for explaining another example of the process of registering the position of the item 60. The item 60 is placed on the shelf 51. An item code D2 is displayed on the shelf 51 near the item 60. The item code D2 is displayed on the shelf 51 with its relative positional relationship with the item 60 determined. The item code D2 is an example of a second item code.

[0121] This method eliminates the need to attach a code to an item, and the benefits of using a two-dimensional code can be enjoyed even when there is no space to attach a code to an item.

[0122] The calculation unit 232 calculates the position of the item code D2 represented by the reference coordinates S1 based on the image of the item code D2 represented in the image representing the vicinity of the item 60 acquired by the camera 13, the distance between the camera 13 and the item code D2, and the current position and current orientation of the terminal 10 calculated by the calculation unit 232, and calculates the position of the item 60 represented by the reference coordinates S1 based on the position of the item code D2 and the positional relationship between the item code D2 and the item 60. The image representing the vicinity of the item 60 preferably includes the item code D2. The image representing the vicinity of the item 60 may or may not include the item 60.

[0123] The camera 13 of the terminal 10 is operated by a user to capture an image including the item code D2. The item code D2 is a two-dimensional code having a rectangular shape. The calculation unit 232 analyzes the image of the item code D2 to obtain information contained in the code, and determines that the item code D2 is a code for calculating the position of the item 60 based on this information.

[0124] The calculation unit 232 uses the third sensor 16 to detect the distance between the camera 13 and the article code D2 located in the direction of the center points of the four position detection patterns of the article code D2 in the image coordinate S3.

[0125] The calculation unit 232 calculates the position of the center point of each of the four position detection patterns of the item code D2 represented by the camera coordinate S2, and then calculates the position of the item 60. There are various possible methods for calculating the position of the item 60 corresponding to the item code D2, but the calculation unit 232 calculates the position of the item 60 represented by the reference coordinate S1, assuming that the item code D2 is affixed at a position relative to the item 60.

[0126] The calculation unit 232 calculates the position of the item 60 represented by the reference coordinates S1 based on the positions of the center points of the four position detection patterns of the item code D2 represented by the reference coordinates S1 and the positional relationship between the item code D2 and the item 60. The positional relationship between the item code D2 and the item 60 is acquired, for example, by analyzing the item code D2.

[0127] For example, if the positions of the center points of the four position detection patterns of the article code D2 expressed in the reference coordinate S1 are A (X1, Y1, Z1), B (X2, Y2, Z2), C (X3, Y3, Z3), and D (X4, Y4, Z4), the relative coordinates A' (X1 + ax, Y1 + ay, Z1), B' (X2 + bx, Y2 + by, Z2), C' (X3 + cx, Y3 + cy, Z3), and D' (X4 + dx, Y4 + dy, Z4) are obtained based on the positional relationship between the article code D2 and the article 60. The plane consisting of these four points A' B' C' D' is the position of the article 60.

[0128] The control unit 231 transmits the positions of the four points representing the item 60 to the server 30 together with an item ID for identifying the item 60, registration information, and specifications of the item code D2. The registration information is a second code. The specifications of the item code D2 are input or selected from a predetermined list by the user operating the user IF 12.

[0129] This registration method is effective when it is not possible to affix the item code D1 to the item 60.

[0130] (Features) The camera 13 of the terminal 10 is operated by a user to capture an image including the item 60. The calculation unit detects the area of ​​the represented item 60 in the image using a classifier.

[0131] The calculation unit 232 has a classifier that identifies an item included in an image. For example, a deep neural network (DNN) that has been trained in advance to detect an item shown in an input image from the image can be used as the classifier. The calculation unit 232 may use a classifier other than a DNN. For example, the calculation unit 232 may use a support vector machine (SVM) that has been trained in advance to input a feature amount (e.g., SIFT, SURF, ORB) calculated from a window set on an image and output a confidence level that an object to be detected is shown in the window. Alternatively, the calculation unit 232 may detect an object region by performing template matching between a template in which an object to be detected is shown and the image.

[0132] When the identifier finds an item to be managed, a point in the area is tapped to calculate the position in the same manner as described above, and the position can be registered as the item's position. In addition, by recording the feature amount in association with the target item in the item management table 200, the item's position can be identified by matching during a search.

[0133] The calculation unit 232 detects the distance between the camera 13 and the article 60 in the direction of the area in the image coordinates S3 where the article is represented, using the third sensor 16. Here, the calculation unit 232 may detect the distance between the camera 13 and the center of gravity of the area in the image coordinates S3 where the article is represented, using the third sensor 16. The distance between the camera 13 and the article 60 in the direction of the area in the image coordinates S3 where the article is represented is an example of the distance between the camera 13 and the vicinity of the article 60.

[0134] The calculation unit 232 calculates the position of the article 60 expressed in the reference coordinates S1 based on the distance between the article 60 and the camera 13 and the current position and current orientation of the terminal 10 calculated by the calculation unit 232.

[0135] Next, the item search process of the item management system 1 will be described below with reference to Fig. 7 and Fig. 8. Fig. 7 is a hardware configuration diagram of the terminal 100 that searches for the item 60. Fig. 8 is a sequence diagram of the item search process of the item management system 1.

[0136] The terminal 100 is used to search for the item 60. The terminal 100 has the same hardware configuration as the terminal 10 described above.

[0137] The terminal 100 includes a communication unit 111, a user interface (IF) 112, a camera 113, a first sensor 114, a second sensor 115, a third sensor 116, an item management unit 117, and the like.

[0138] The item management unit 117 includes a control unit 1231 , a calculation unit 1232 , a determination unit 1233 , and a generation unit 1234 .

[0139] The communication unit 111 corresponds to the communication unit 11, the IF 112 corresponds to the IF 12, the camera 113 corresponds to the camera 13, the first sensor 114 corresponds to the first sensor 14, the second sensor 115 corresponds to the second sensor 15, the third sensor 116 corresponds to the third sensor 16, and the item management unit 117 corresponds to the item management unit 17. The control unit 1231 corresponds to the control unit 231, and the calculation unit 1232 corresponds to the calculation unit 232.

[0140] Here, IF 112 is an example of a display unit, camera 13 is an example of a second image acquisition unit, first sensor 114 is an example of a second first sensor, and second sensor 15 is an example of a second second sensor.

[0141] The above-mentioned explanations of the communication unit 11, user interface (IF) 12, camera 13, first sensor 14, second sensor 15, third sensor 16, and item management unit 17 of the terminal 10 are appropriately applied to the communication unit 111, user interface (IF) 112, camera 113, first sensor 114, second sensor 115, third sensor 116, and item management unit 117 of the terminal 100. In addition, the above-mentioned explanations of the control unit 231 and calculation unit 232 of the item management unit 17 of the terminal 10 are appropriately applied to the control unit 1231 and calculation unit 1232 of the item management unit 117.

[0142] The user uses the terminal 100 to search for an article registered in the article management table 200 stored in the storage device 32 of the server 30 .

[0143] The processes from steps S201 to S204 are similar to the processes from steps S101 to S104 described above.

[0144] After the position and orientation of the terminal 100 expressed by the reference coordinates S1 are calculated (step S204), the control unit 1231 transmits a search request for information on the item to be searched for to the server 30 via the macrocell base station 41 and the communication network 40 (step S205). The search request preferably includes a code ID representing the reference code C1.

[0145] Next, in response to the search request, the search unit 343 of the server 30 refers to the item management table 200 stored in the storage device 32 and generates search information including a list of items whose positions are registered at the reference coordinates S1. The search information includes information such as item IDs, names, and states. The search unit 343 transmits the search information to the terminal 100 via the communication network 40 and the macrocell base station 41 (step S206).

[0146] Next, the control unit 1231 of the terminal 100 displays the search information on the user IF 112. The user selects an item to be searched for by operating the user IF 112. The control unit 1231 transmits a location request for the location of the selected item to the server 30 via the macrocell base station 41 and the communication network 40 (step S207).

[0147] Next, in response to the location request, the search unit 343 of the server 30 refers to the item management table 200 stored in the storage device 32, and transmits location information and registration information indicating the location of the item to be searched for to the terminal 100 via the communication network 40 and the macrocell base station 41 (step S208).

[0148] Next, the user moves to the item 60 to be searched for together with the terminal 100 with the camera 113 activated (step S209). The environment inside the warehouse 50 captured by the camera 113 is displayed on the user IF 112. The generation unit 1234 calculates the direction from the position of the terminal 100 to the item 60 based on the position and orientation of the terminal 100 expressed in the reference coordinate system S1 and the position of the item 60 to be searched for. The generation unit 1234 generates a display that guides the user IF 112 to the position of the item 60. The control unit 1231 may display the display generated by the generation unit 1234 superimposed on the environment inside the warehouse 50.

[0149] Next, the user captures images of the multiple items 60 arranged on the shelf 51 using the camera 113 of the terminal 100 (step S210).

[0150] Next, if the position of the searched article 60 is included in the field of view of the camera 113, the control unit 1231 displays article information representing the searched article 60 on the user IF 112 (step S211). The determination unit 1233 determines whether or not the position of the searched article 60 expressed in the reference coordinates S1 is included in the field of view of the camera 113. The determination unit 1233 determines whether or not the position of the searched article 60 is included in the field of view of the camera 113 based on internal parameters including the focal length and field of view of the camera 113, external parameters including the mounting position of the terminal 100 of the camera 113, the position and orientation of the terminal 100, and the position of the article 60.

[0151] The generating unit 1234 may generate the product information based on the registration information. For example, if the registration information is one-point tap, the generating unit 1234 may generate the product information having a spherical figure, and if the registration information is four-point tap, the generating unit 1234 may generate the product information having a planar figure. Note that other expressions may be used as the product display information.

[0152] The generation unit 1234 places the item information at the position of the searched item 60 represented by the reference coordinates S1, projects the item information onto a plane represented by the image coordinates S3 using viewpoint projection transformation, and generates an extended image in which the item information is represented so as to overlap with the searched item 60. This allows the user to recognize that he or she has found the searched item 60.

[0153] Next, the user operates the user IF 112 to input information indicating that the searched item 60 has been found to the terminal 100. The control unit 1231 of the terminal 100 transmits the discovery information to the server 30 via the macrocell base station 41 and the communication network 40 (step S212). The discovery information includes, for example, information indicating that the searched item 60 has been found, an item ID, and the like.

[0154] Next, the search unit 343 of the server 30 changes the information in the status column 206 associated with the discovered item from "unsearched" to "searched" in the item management table 200 stored in the storage device 32 in response to the discovery information, and registers it (step S213), thereby completing the series of processes.

[0155] According to the item management system 1 of the present embodiment described above, the position of an item is represented by reference coordinates with the reference code displayed in the warehouse as the origin, so that the actual position of an item can be correctly calculated even in a warehouse where GPS information cannot be obtained.

[0156] In the above embodiment, the positions of all items are expressed with one reference code as the origin. However, if the warehouse is large, the positions of items may be managed using auxiliary coordinates with the origin at the auxiliary code C2, which is determined by the relative positional relationship with the reference code C1.

[0157] In the example shown in Fig. 1, an auxiliary code C2 is displayed on each of the multiple shelves 51. In each shelf 51, the positions of the multiple items 60 arranged on this shelf 51 are represented by auxiliary coordinates S4 with the auxiliary code C2 displayed on that shelf 51 as the origin. The positions of the items 60 represented by the auxiliary code C2 are converted to positions in the reference coordinates S1 using predetermined matrices MW2 to MW4. The explanation of the reference coordinates S1 is appropriately applied to the auxiliary coordinates S4.

[0158] 4(A) registers three auxiliary codes C2 displayed on three shelves 51. The three auxiliary codes C2 are identified by different code IDs ref2 to ref4. For example, the calculation unit 232 converts the position of the item represented by the auxiliary coordinates S4 of the auxiliary code C2 of ref2 into the position of the reference coordinates S1 using the matrix MW2.

[0159] The matrix MWn (n is an integer representing a management number that identifies the fixed location code) is expressed by the following formula 12. Here, MW1 is a matrix that converts a position expressed by coordinates SA, whose origin is the camera position of camera 13 when camera 13 is started, into a position of reference coordinates S1. Mw2 is a matrix that converts a position expressed by coordinates SA, whose origin is the camera position of camera 13 when camera 13 is started, into a position expressed in a coordinate system whose origin is auxiliary code C2.

[0160]

number

[0161] In the code management table 100, only the values ​​that vary among the matrix components may be registered in the transformation matrix column 105. In addition, when the matrix MWn does not include transformation components other than rotation and translation, the matrix MWn may be divided into matrices that represent rotation and translation. Here, the matrix that represents rotation may be represented by Euler angles that can be expressed by three components.

[0162] Furthermore, for an item whose position is calculated using auxiliary coordinates S4 with the auxiliary code C2 as the origin, the position of this item expressed in a coordinate system with the auxiliary code C2 as the origin may be registered in the item management table 200. In this case, it is preferable to create a new reference code column in the item management table 200 so that it is possible to identify whether the position of the item is expressed in reference coordinates S1 with the reference code C1 as the reference, or in auxiliary coordinates S4 with the auxiliary code C2 as the reference.

[0163] In the above-described item search process, the user can quickly perform the task of searching for items placed on the shelves 51 by photographing the auxiliary code C2 displayed on each shelf 51 using the terminal 10.

[0164] For example, in step S206 of the above-mentioned item search process, the search unit 343 of the server 30 may refer to the item management table 200 stored in the storage device 32 in response to a search request, and generate item information including a list of items whose positions are registered at the auxiliary coordinates S4 of the auxiliary code C2. Only the item information of items associated with the auxiliary code C2 is displayed on the terminal 100. This can reduce the amount of information processed in the item management system 1.

[0165] Furthermore, the auxiliary code C2 can also be used to correct the position of the terminal 100 within the warehouse 50. The position of the terminal 100 in the reference coordinate system S1 when the image of the auxiliary code C2 is acquired is calculated by the calculation unit 1232. The position of the terminal 100 calculated by the calculation unit 1232 may be compared with the position of the terminal 100 calculated based on the position of the auxiliary code registered in the code management table 100, to correct the position of the terminal 100 within the warehouse 50.

[0166] In the code management table 100, the auxiliary code C2 identified by ref2 to ref4 is displayed on a fixed shelf 51 in the warehouse 50, and is therefore registered as "fixed" in the type column 103. The position and orientation of the auxiliary code C2 are registered in the position column 104 of the code management table 100. The position is expressed in reference coordinates S1, and the orientation is expressed in Euler angles.

[0167] Meanwhile, a movable cart 52 is placed in the warehouse 50, and an item 60 is also placed on this cart. An auxiliary code C3 is displayed on the cart 52. The position and orientation of the auxiliary code C3 change as the cart 52 moves.

[0168] In the code management table 100 shown in FIG. 4(A), the auxiliary code C3 identified by ref5 changes position and direction as the cart 52 moves, so it is registered as "indefinite" in the type column 103.

[0169] The calculation unit 1232 uses the matrix MW5 to convert the position of the article represented by the auxiliary coordinates S4 with the auxiliary code C3 of ref5 as the origin into the position of the reference coordinates S1. The matrix MW5 is also represented by the above formula 12. This matrix MW5 is updated every time the cart 52 moves.

[0170] Next, an example will be described in which the position of an item is calculated using two auxiliary codes C2 (hereinafter also referred to as auxiliary codes C2-1 and C2-2) fixed to different shelves 51. Assume that the camera 113 of the terminal 100 first acquires an image of the auxiliary code C2-1, and then acquires an image of the item 60 whose position is represented by auxiliary coordinates S3 with the auxiliary code C2-1 as the origin.

[0171] Let MW3 be the matrix that converts the position of the article 60 represented by the auxiliary coordinates S3 with the auxiliary code C2-1 as the origin into the position of the reference coordinates S1. Let Mw3 be the matrix that converts the position represented by the coordinates SA with the camera position of the camera 113 at the time the camera 113 is started as the origin into the position represented by the coordinates with the auxiliary code C2-1 as the origin.

[0172] Also, let MW4 be the matrix that converts the position of the item 60 represented in the auxiliary coordinate system S4 with the auxiliary code C2-2 as the origin into the position of the reference coordinate S1. Let M2 be the matrix that converts the position represented by the coordinate SA with the position of the camera 113 at the time the camera 113 is started as the origin into the position of the camera coordinate S2 of the camera 113 of the terminal 100 located at an arbitrary position within the warehouse 50. Here, the arbitrary position within the warehouse 50 is the position of the camera 113 when the image of the item 60 was captured.

[0173] A matrix Mc4 that converts the position of the article 60 represented in the auxiliary coordinate system S4 with the auxiliary code C2-2 as the origin into the position of the camera coordinates S2 of the camera 113 when the image of the article 60 was captured is expressed by the following formula 13.

[0174]

number

[0175] Next, an example of calculating the position of an item using the auxiliary code C2 fixed to the shelf 51 and the auxiliary code C3 displayed on the cart 52 will be described below. Assume that the camera 113 of the terminal 100 first acquires an image of the auxiliary code C2 displayed near the cart 52, and then acquires an image of the item 60 whose position is represented by auxiliary coordinates S3 with the auxiliary code C3 as the origin.

[0176] Let MW5 be the matrix that converts the position of the article 60 represented by the auxiliary coordinates S4 with the auxiliary code C2 as the origin into the position of the reference coordinates S1. Let Mw5 be the matrix that converts the position represented by the coordinates SA with the camera position of the camera 113 at the time the camera 113 is started as the origin into the position of the auxiliary coordinates S4 with the auxiliary code C2 as the origin.

[0177] Also, let Mw6 be the matrix that transforms the position represented by the coordinate SA, which has the camera position of the camera 113 at the time the camera 113 is started as the origin, into the position of the auxiliary coordinate S4, which has the auxiliary code C3 as the origin.

[0178] A matrix MW6 that converts the position of the auxiliary coordinate S4, which has the auxiliary code C3 as its origin, into the position of the reference coordinate S1 is expressed by the following formula 14.

[0179]

number

[0180] The user can easily manage the position of the item 60 placed on the cart 52 by first acquiring an image of the auxiliary code C2 displayed near the cart 52 using the terminal 100, and then acquiring an image of the auxiliary code C3 displayed on the cart 52 using the terminal 100.

[0181] In the present invention, the item location registration device, the item management system, and the item search device of the above-mentioned embodiments can be modified as appropriate without departing from the spirit of the present invention. Furthermore, the technical scope of the present invention is not limited to those embodiments, but extends to the inventions described in the claims and their equivalents.

[0182] For example, in an item position registration device, the calculation unit may calculate a first position and a first orientation of the position calculation device expressed in a reference coordinate system having an origin at the information code based on an image of the information code acquired by the image acquisition unit and the distance between the image acquisition unit and the information code, calculate a second position representing the current position of the position calculation device expressed in the reference coordinate system and a second orientation representing the current orientation based on the calculated first position and first orientation, position information and angle information, and calculate the position of the item expressed in the reference coordinate system based on an image representing the vicinity of the item acquired by the image acquisition unit, the distance between the image acquisition unit and the vicinity of the item, and the second position and second orientation.

[0183] In the above embodiment, the server stores the code management table and the item management table, but the terminal may store the code management table and the item management table. In the above embodiment, the terminal that registers the location of an item and the terminal that searches for the item are different, but one terminal having the functions of the two terminals may be used to register the location of an item and to search for the item.

[0184] In the above-described embodiment, the LiDAR sensor is used as the distance measurement unit, but other sensors may be used as the distance measurement unit. For example, a stereo camera, an RGB-D sensor, or a distance sensor may be used as the distance measurement unit. [Explanation of symbols]

[0185] 1. Inventory management system 10 Terminal 11 Communications Department 12 User Interface 13. Camera 14 First Sensor 15 Second Sensor 16 3rd Sensor 17 Materials Management Department 18 Signal Line 21 Communication Interface 22 Memory 23 Processors 231 Control Unit 232 Calculation section 233 Generation part 30 Servers 31 Communication Interface 32 Storage Device 33 Memory 34 processors 341 Control Unit 342 Registration Department 343 Search Department 35 Signal Line 40 Communication Network 41 Base station 50 Warehouse 51 Shelf 52 Trolley 60 Goods C1 Reference Code C2 Auxiliary Code C3 Auxiliary Code D1 Item Code D2 Item Code 100 devices 111 Communications Department 112 User Interface 113 Camera 114 First Sensor 115 Second Sensor 116 3rd Sensor 117 Materials Management Department 118 Signal Line 121 Communication Interface 122 Memory 123 processors 1231 Control Unit 1232 Calculation section 1233 Judgment section 1234 Generation part

Claims

1. An article position registration device that calculates an article position relative to an information code, An image acquisition unit; a first sensor that detects position information for determining a position change amount; a second sensor that detects angle information for determining an amount of change in angle; a first calculation unit that calculates a first position and a first orientation of the article position registration device expressed in a reference coordinate system having an origin at the information code, based on an image of the information code acquired by the image acquisition unit and a distance between the image acquisition unit and the information code; a second calculation unit that calculates a second position representing a current position of the item position registration device expressed in the reference coordinates and a second orientation representing a current orientation based on the first position and the first orientation calculated by the first calculation unit, the position information detected by the first sensor, and the angle information detected by the second sensor, and calculates the position of the item expressed in the reference coordinates based on an image representing the vicinity of the item acquired by the image acquisition unit, a distance between the image acquisition unit and the vicinity of the item, and the second position and the second orientation; An article position registration device comprising:

2. The article position registration device according to claim 1 , wherein the second calculation unit calculates the position of the article expressed in the reference coordinates based on one point selected on an image representing a vicinity of the article.

3. The article position registration device according to claim 1 , wherein the second calculation unit calculates, as the position of the article, a plane represented by the reference coordinates defined based on at least three points selected on an image representing a vicinity of the article.

4. a first item code disposed on the item; The item position registration device of claim 1, wherein the second calculation unit calculates a plane containing the first item code expressed in the reference coordinates based on an image of the first item code shown in the image acquired by the image acquisition unit, a distance between the image acquisition unit and the first item code, and the second position and the second orientation calculated by the second calculation unit, and a specified area within the plane containing the first item code is set to the position of the item expressed in the reference coordinates.

5. a second item code having a first positional relationship relative to the item is disposed near the item; The item position registration device of claim 1, wherein the second calculation unit calculates the position of the second item code expressed in the reference coordinates based on an image of the second item code represented in the image acquired by the image acquisition unit, the distance between the image acquisition unit and the second item code, and the second position and the second orientation calculated by the second calculation unit, and calculates the position of the item expressed in the reference coordinates based on the position of the second item code and the first positional relationship.

6. a third sensor capable of detecting a distance between the vicinity of the item and the item position registration device; 2. The object position registration device according to claim 1, wherein the second calculation unit calculates the position of a plane surrounded by points on the object that are measured by the third sensor and are at the same distance from the vicinity of the object to the image acquisition unit as the position of the object expressed in the reference coordinates.

7. A storage unit that stores a feature amount of the article shown in an image, 2. The item position registration device according to claim 1, wherein the second calculation unit detects an area in which the item is represented from the image acquired by the image acquisition unit based on features of the item stored in the memory unit, and calculates the position of the item expressed in the reference coordinates based on the detected area, a distance between the image acquisition unit and a vicinity of the item, and the second position and the second orientation calculated by the second calculation unit.

8. 2. The object position registration device according to claim 1, further comprising a third calculation unit that calculates the position of the object represented in auxiliary coordinates having an origin in an auxiliary code having a second positional relationship relative to a reference code, based on the second positional relationship.

9. An article position registration device that calculates an article position relative to a reference code, An image acquisition unit; a first sensor that detects position information for determining a position change amount; a second sensor that detects angle information for determining an amount of change in angle; a first calculation unit that calculates a first position and a first orientation of the article position registration device expressed in a reference coordinate system having an origin at the information code, based on an image of the information code acquired by the image acquisition unit and a distance between the image acquisition unit and the information code; a second calculation unit that calculates a second position representing a current position of the item position registration device expressed in the reference coordinates and a second orientation representing a current orientation based on the first position and the first orientation calculated by the first calculation unit, the position information detected by the first sensor, and the angle information detected by the second sensor, and calculates the position of the item expressed in the reference coordinates based on an image representing the vicinity of the item acquired by the image acquisition unit, a distance between the image acquisition unit and the vicinity of the item, and the second position and the second orientation; The article position registration device having a storage unit that stores the position of the item represented by the reference coordinates calculated by the item position registration device; An article management system comprising:

10. An article search device that searches for an article whose position is calculated by the article position registration device according to claim 1, A display unit; A second image acquisition unit; a second first sensor that detects position information for determining a position change amount; a second sensor that detects angle information for determining an amount of change in angle; a determination unit that determines whether the position of the article represented by the reference coordinates is included in the field of view of the second image acquisition unit; a generation unit that generates an extended image in which article information related to the article is superimposed on an image acquired by the second image acquisition unit when the determination unit determines that the position of the article represented by the reference coordinates is included in the field of view of the second image acquisition unit; and a control unit that displays the extended image generated by the generation unit on the display unit; An object searching device comprising:

Images