Article search device and article management system

Abstract

To provide an article search device with which it is possible to search for an article even in a place where GPS information cannot be acquired.SOLUTION: The article search device determines whether the registration position of an article expressed by a reference coordinate system in which an information code is adopted as origin is included in a visual field of an image acquisition unit, determines whether the registration position of the article matches the current position expressed by the reference coordinate when the registration position of the article is included in the visual field of the image acquisition unit, and locates article information at the position determined based on the registration position of the article in the image acquired by the image acquisition unit and generates an extended image that is displayed on a display unit when the registration position of the article matches the current position.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an item search device and an item management system. [Background technology]

[0002] 2. Description of the Related Art Conventionally, augmented reality technology is used to place an augmented reality object on an image that represents a real object on the ground.

[0003] For example, it has been proposed to use an augmented reality technology to provide route guidance to reach a destination (see, for example, Patent Document 1). In Patent Document 1, the position of a real object on the ground is expressed in world coordinates (also called global coordinates) using GPS (Global Positioning System) information. The world coordinates have their origin at a predetermined position on the ground. The GPS information is acquired by receiving radio waves from satellites.

[0004] Here, an augmented reality object is displayed on the image of the terminal in correspondence with a real object on the ground. The position of the real object is calculated based on the position and orientation of the terminal estimated using GPS information, but an error may occur.

[0005] Patent Document 1 proposes using AR anchors placed at predetermined positions on the ground together with GPS information to correct the position of a terminal and calculate the correct position of a real-world object. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] International Publication No. 2020 / 227734 Summary of the Invention [Problem to be solved by the invention]

[0007] Meanwhile, it is desirable to use augmented reality technology even in areas where GPS information cannot be acquired or received, such as indoors or underground.

[0008] In order to use augmented reality technology, it is preferable that the positions of real objects be calculated correctly even indoors where GPS information cannot be obtained.

[0009] Then, when searching for an object whose location has been registered, an augmented reality object is displayed based on the location of the registered object.

[0010] However, if the object is a movable item, the position of the item at the time of searching may have moved from the position of the item at the time of registration.

[0011] In such a situation, when an augmented reality object is displayed on the terminal image corresponding to the position of the item at the time of registration, it may happen that the position of the augmented reality object does not match the position of the item shown in the image.

[0012] The same can happen if there is a large error in the terminal's calculation of its own position, or if another object is placed in front of the item being searched for.

[0013] If the position of the augmented reality object in the image on the terminal does not match the position of the item depicted in the image, the user may not be able to find the item or may confuse the item if multiple items are lined up next to each other.

[0014] Therefore, an object of the present disclosure is to provide an item search device that is capable of searching for items even in places where GPS information cannot be obtained. [Means for solving the problem]

[0015] According to one embodiment, an item search device is provided, which searches for the position of an item represented in a reference coordinate system with an information code as the origin, and is characterized by having a display unit, an image acquisition unit, a first determination unit that determines whether or not the registered position of the item represented in the reference coordinates registered in the item management table is included in the field of view of the image acquisition unit, a second determination unit that determines whether or not the registered position of the item and the current position of the item represented in the reference coordinates match when the first determination unit determines that the registered position of the item is included in the field of view of the image acquisition unit, and a generation unit that, when the second determination unit determines that the registered position of the item and the current position of the item match, arranges item information related to the item at a position determined based on the registered position of the item in the image acquired by the image acquisition unit to generate an extended image to be displayed on the display unit.

[0016] According to another embodiment, an item management system is provided, which is an item search device that searches for a position of an item represented in a reference coordinate system with an information code as an origin, and includes a display unit, an image acquisition unit, a first determination unit that determines whether a registered position of the item represented in reference coordinates registered in an item management table is included in a field of view of the image acquisition unit, a second determination unit that determines whether a registered position of the item and a current position of the item represented in the reference coordinates match when the first determination unit determines that the registered position of the item is included in the field of view of the image acquisition unit, and a generation unit that, when the second determination unit determines that the registered position of the item and the current position of the item match, arranges item information about the item at a position determined based on the registered position of the item in an image acquired by the image acquisition unit to generate an extended image to be displayed on the display unit, and a storage unit that stores the item management table. Effect of the Invention

[0017] The item search device according to the present invention has the advantage of being able to search for an item even in a place where GPS information cannot be obtained. [Brief description of the drawings]

[0018] [Figure 1] 1 is a schematic configuration diagram of an item management system in which a terminal including an item management unit is implemented. [Diagram 2] FIG. 11 is a sequence diagram of an item location registration process in the item management system. [Diagram 3] FIG. 2 is a hardware configuration diagram of a terminal that registers the position of an item. [Figure 4] FIG. 1A is a diagram showing a code management table, and FIG. 1B is a diagram showing an article management table. [Diagram 5] FIG. 11 is a diagram illustrating a process for calculating the position of a terminal. [Figure 6] FIG. 4A is a diagram for explaining an example of a process for registering the position of an item, and FIG. 4B is a diagram for explaining another example of a process for registering the position of an item. [Figure 7] FIG. 2 is a hardware configuration diagram of a terminal that searches for an item. [Figure 8] FIG. 11 is a sequence diagram of an item search process in the item management system. [Figure 9] 13 is an example of an operational flowchart of a terminal search process according to a first embodiment. [Figure 10] 13 is an example of an operational flowchart of a terminal search process according to a second embodiment. [Figure 11] 13 is an example of an operational flowchart of a terminal search process according to a third embodiment. [Figure 12] 13 is an example of an operational flowchart of a terminal search process according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Fig. 1 is a schematic configuration diagram of an item management system 1 having a terminal 10 in which an item management unit 17 is implemented and a terminal 100 in which an item management unit 117 is implemented. Fig. 2 is a sequence diagram of an item position registration process of the item management system 1. Fig. 3 is a hardware configuration diagram of the terminal 10.

[0020] The item management system 1 has at least one terminal 10, at least one terminal 100, and a server 30. An item management unit 17 is implemented in the terminal 10 that registers the location of an item. An item management unit 117 is implemented in the terminal 100 that searches for an item. The terminals 10, 100 are connected to the server 30 via the macrocell base station 41 and the communication network 40, for example, by accessing a wireless base station 41 (hereinafter also referred to as a macrocell base station 41) that provides a macrocell that is connected to the communication network 40 to which the server 30 is connected via a gateway (not shown) or the like. The terminal 10 is an example of an item location registration device. The terminal 100 is an example of an item search device.

[0021] 1 shows only one terminal 10 and one terminal 100, the article management system 1 may have a plurality of terminals 10 and 100. Similarly, a plurality of macrocell base stations 41 may be connected to the communication network 40. Below, a process of registering the position of an article using the terminal 10 will be described. After that, a process of searching for an article using the terminal 100 will be described.

[0022] A user (not shown) of the item management system 1 manages the positions of multiple items 60 placed in a warehouse 50 using a terminal 10 that registers the positions of the items. For example, the user uses the terminal 10 to register the positions of the items 60 in the warehouse 50 in the server 30. It is not always possible to receive GPS information inside the warehouse 50. Therefore, the item management system 1 manages the positions of the items 60 without using GPS information, as described below.

[0023] Next, the item location registration process of the item management system 1 will be described below with reference to Fig. 2. First, the user uses the camera 13 of the terminal 10 to capture an image of the reference code C1 displayed near the entrance of the warehouse 50 (step S101). The vicinity here refers to an area in which the item, the reference code C1, the item code D1, the item 60, etc. can be identified. The reference code C1 is an example of an information code.

[0024] The reference code C1 represents the origin of a reference coordinate S1 that represents the position of the item 60 within the warehouse 50. It is preferable to select the entrance / exit of the warehouse 50 as the position where the reference code C1 is displayed, from the viewpoint that a user who enters the warehouse 50 can obtain an image of the reference code C1.

[0025] The reference code C1 may be displayed on a medium such as paper, etc. Also, the reference code C1 may be displayed on a display device.

[0026] The reference code C1 may be a code having information such as a one-dimensional code or a two-dimensional code. For example, the code described in Japanese Patent No. 5908113 may be used as the reference code C1.

[0027] Next, the terminal 10 analyzes the image of the reference code C1 to obtain the code ID contained in the code, and transmits this code ID to the server 30 via the macrocell base station 41 and the communication network 40 (step S102).

[0028] Next, the server 30 receives the code ID and transmits the dimensions of the reference code C1 associated with this code ID to the terminal 10 via the communication network 40 and the macrocell base station 41 (step S103). For example, when the reference code C1 is a rectangular two-dimensional code, the dimensions of the reference code C1 represent the vertical and horizontal lengths.

[0029] Next, the terminal 10 calculates the position and orientation of the terminal 10 expressed in reference coordinates S1 having the origin at the reference code C1 based on the image showing the reference code C1 and the distance between the camera 13 of the terminal 10 and the reference code C1 (step S104). The process of calculating the position of the terminal 10 will be described in detail later.

[0030] Next, with the camera 13 activated, the user moves together with the terminal 10 to the location of the item 60 whose location is to be registered (step S105).

[0031] Next, the user acquires an image showing the vicinity of the item 60 using the camera 13 of the terminal 10 (step S106). The image showing the vicinity of the item 60 is an image showing the item or an image showing an item code. The vicinity here refers to a range in which the user can visually recognize that the item is the target item, including the position of the item itself. The terminal 10 has a first sensor 14 that detects position information for determining the amount of change in position of the terminal 10, and a second sensor 15 that detects angle information for determining the amount of change in angle of the terminal 10. The amount of movement of the terminal 10 from the position where the image of the reference code C1 was acquired is obtained by accumulating the position information detected by the first sensor 14. In addition, the amount of change in orientation of the terminal 10 from the orientation when the image of the reference code C1 was acquired is obtained by accumulating the angle information detected by the second sensor 15.

[0032] Next, the terminal 10 calculates the current position and orientation of the terminal 10 based on the position and orientation when the image of the reference code C1 was acquired, the position information detected by the first sensor 14, and the angle information detected by the second sensor 15. The terminal 10 also calculates the position of the item 60 represented by the reference coordinates S1 based on the current position and orientation of the terminal 10, the image of the vicinity of the item 60 acquired by the camera 13, and the distance between the camera 13 and the vicinity of the item 60 (step S107). Details of these processes of the terminal 10 will be described later.

[0033] Next, the terminal 10 transmits the position of the item 60 expressed in the reference coordinates S1 together with the item ID to the server 30 via the macrocell base station 41 and the communication network 40 (step S108).

[0034] Next, the server 30 registers the position of the item 60 represented by the reference coordinates S1 in the item management table 200 (see FIG. 4(B)), and ends the series of processes.

[0035] According to the item management system 1 of this embodiment described above, the position of the item 60 is represented by a reference coordinate S1 with the reference code C1 displayed in the warehouse 50 as the origin, so that the actual position of the item can be correctly calculated even in a warehouse 50 where GPS information cannot be obtained.

[0036] The article management system 1 may include multiple terminals 10, but since each terminal has the same configuration and executes the same processes with respect to article management processing, the following description will focus on one terminal 10.

[0037] Next, the terminal 10 in which the item management unit 17 is implemented will be described below with reference to Fig. 3. Fig. 3 is a hardware configuration diagram of the terminal 10 that registers the position of an item 60. The terminal 10 has a communication unit 11, a user interface (IF) 12, a camera 13, a first sensor 14, a second sensor 15, a third sensor 16, an item management unit 17, etc.

[0038] The communication unit 11, the user IF 12, the camera 13, the first sensor 14, the second sensor 15, the third sensor 16, and the item management unit 17 are communicatively connected via a signal line 18.

[0039] The communication unit 11 is an example of a terminal communication unit, and has an interface circuit for connecting the terminal 10 to the macrocell base station 41. The communication unit 11 is configured to be able to communicate with the server 30 via the macrocell base station 41 and the communication network 40.

[0040] The user IF 12 displays processing information associated with processing of the terminal 10, and inputs operation information associated with user operations. The processing information includes, for example, an image captured by the camera 13. The user IF 12 includes, for example, a touch panel.

[0041] Camera 13 is an example of an image acquisition unit. Camera 13 is attached to terminal 10 so as to face outward from terminal 10. Camera 13 outputs the acquired image to item management unit 17 via signal line 18. Camera 13 has a two-dimensional detector configured with an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS, and an imaging optical system that forms an image of the area to be photographed on the two-dimensional detector.

[0042] The first sensor 14 detects position information for determining the amount of change in position of the terminal 10, and outputs this position information to the item management unit 17 via a signal line 18. As the first sensor 14, for example, an acceleration sensor can be used.

[0043] The second sensor 15 detects angle information for determining the amount of change in angle of the terminal 10, and outputs this angle information to the item management unit 17 via a signal line 18. As the second sensor 15, for example, a gyroscope can be used.

[0044] The third sensor 16 is an example of a distance measuring unit. The third sensor 16 detects the distance between the item 60 and the terminal 10, and outputs the distance to the item management unit 17 via the signal line 18. As the third sensor 16, for example, a LiDAR sensor capable of three-dimensional measurement can be used.

[0045] The article management unit 17 executes control processing and calculation processing. To this end, the terminal 10 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the terminal 10 to the signal line 18.

[0046] The memory 22 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 22 stores computer programs of applications used in the information processing executed by the processor 23 and various data.

[0047] All or part of the functions of the terminal 10 are functional modules realized by, for example, a computer program running on the processor 23. The processor 23 has a control unit 231 and a calculation unit 232. The calculation unit 232 is an example of a first calculation unit, a second calculation unit, and a third calculation unit. The functional modules of the processor 23 may be dedicated arithmetic circuits provided in the processor 23. The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical calculation unit, or a graphic processing unit. Details of the operation of the terminal 10 will be described later.

[0048] The server 30 executes a control process, an update process, and a search process. To that end, the server 30 has a communication interface (IF) 31, a storage device 32, a memory 33, and a processor 34. The communication IF 31, the storage device 32, and the memory 33 are connected to the processor 34 via a signal line 35. The server 30 may further have input devices such as a keyboard and a mouse, and a display device such as a liquid crystal display.

[0049] The communication IF 31 has an interface circuit for connecting the server 30 to the communication network 40. The communication IF 31 is configured to be able to communicate with the terminal 10 via the communication network 40 and the macrocell base station 41. The communication IF 31 is an example of a server communication unit.

[0050] The storage device 32 includes, for example, a hard disk device or an optical recording medium and an access device therefor. The storage device 32 stores a code management table 100 (see FIG. 4(A)) for managing the reference code C1 and the like, and an item management table 200 (see FIG. 4(B)). The storage device 32 may further store identification information for identifying the terminal 10. Furthermore, the storage device 32 may store a computer program for executing the processing of the server 30 related to the item management processing, which is executed on the processor 34. The storage device 32 is an example of a memory unit.

[0051] 4A is a diagram showing a code management table 100. The code management table 100 has a management number column 101, a code ID column 102, a type column 103, a position column 104, a transformation matrix column 105, and a specification column 106.

[0052] In the management number field 101, a management number for managing a code managed by the article management system 1 is registered. In the code ID field 102, a code ID for identifying the code is registered. In the type field 103, a type of the code is registered. There are three types of codes: an origin code representing the origin of the reference coordinate S1, such as the reference code C1; a fixed code representing the origin of a local auxiliary coordinate fixed in the warehouse; and an indefinite code representing the origin of a local auxiliary coordinate movable in the warehouse. The fixed code and the indefinite code will be described later. In the position field 104, a position of the code represented by the reference coordinate S1 is registered. In the transformation matrix field 105, a transformation matrix for transforming a position represented by the auxiliary coordinate to a position represented by the reference coordinate S1 is registered. In the specification field 106, the dimensions, version, error correction rate, etc. of the code are registered.

[0053] 4(B) is a diagram showing an item management table 200. In the item management table 200, information regarding the location of the item 60 placed in the warehouse 50 is registered. The item management table 200 has a management number column 201, an item ID column 202, a registration information column 203, a location column 204, a specification column 205, and a status column 206. The item management table 200 may also register the name of the item.

[0054] The management number column 101 registers a management number for managing an item managed by the item management system 1. The item ID column 202 registers an item ID for identifying an item. The registration information column 203 registers registration information indicating a method for registering an item's position; the registration information will be described later. The position column 204 registers information indicating the position of the item expressed in the reference coordinates S1. In the specification column 205, when the position of the item is represented using an item code, the dimensions, version, error correction rate, etc. of the item code are registered. The status column 206 registers whether the item is in a found state or an unsearched state.

[0055] The memory 33 includes, for example, a volatile semiconductor memory and a non-volatile semiconductor memory. The memory 33 stores computer programs of applications and various data used in information processing executed by the processor 34. Note that the restriction information may be stored in the memory 33. The memory 33 is an example of a storage unit.

[0056] All or part of the functions of the server 30 are functional modules realized by, for example, a computer program running on the processor 34. The processor 34 has a control unit 341, a registration unit 342, and a search unit 343. Alternatively, the functional modules of the processor 34 may be dedicated arithmetic circuits provided in the processor 34. The processor 34 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 34 may further have other arithmetic circuits such as a logic arithmetic unit, a numerical arithmetic unit, or a graphic processing unit.

[0057] When the control unit 341 receives a code ID from the terminal 10, it refers to the code management table 100 stored in the storage device 32, reads out the dimensions associated with this code ID, and transmits them to the terminal 10.

[0058] When the registration unit 342 receives the position and the item ID of the item 60 from the terminal 10, it associates the position of the item 60 with the item ID of the item 60 and registers it in the item management table 200 stored in the storage device 32. Details of the operation of the search unit 343 will be described later.

[0059] Next, the process of calculating the position of the terminal 10 in the above-mentioned step S104 will be described below. The calculation unit 232 of the terminal 10 calculates the first position P1 and the first orientation A1 of the terminal 10 expressed in the reference coordinates S1 having the origin at the reference code C1, based on the image showing the reference code C1 and the distance between the camera 13 of the terminal 10 and the reference code C1.

[0060] The calculation unit 232 uses the third sensor 16 to calculate the distance between the third sensor 16 and the reference code C1. The positional relationship between the third sensor 16 and the camera 13 is stored as an internal parameter in the memory 22. The calculation unit 232 calculates the distance between the camera 13 and the reference code C1 based on the distance between the third sensor 16 and the reference code C1 and the internal parameter.

[0061] Furthermore, the calculation unit 232 receives the dimensions of the reference code C1 from the server 30. The reference code C1 has a square outer shape. The vertical dimension and the horizontal dimension of the reference code C1 are the same. If the reference code C1 includes the dimensions of the reference code C1 as information, the calculation unit 232 may obtain the dimensions of the reference code C1 by analyzing an image of the reference code C1.

[0062] The calculation unit 232 may also calculate the distance between the camera 13 and the reference code C1 based on the dimensions of the reference code C1 and the number of pixels expressing the reference code C1 shown in the image.

[0063] 5 is a diagram for explaining the process of calculating the position of the terminal 10. The camera coordinate S2 having an origin O2 on the lens of the camera 13 has a Z axis in the direction of the optical axis of the lens of the camera 13, and has an X axis and a Y axis that are orthogonal to each other at the origin O2 of this Z axis.

[0064] The reference code C1 has position detection patterns at the four corners of a square. In the camera coordinate system S2, the center points of the four position detection patterns of the reference code C1 are represented as A (X1, Y1, Z1), B (X2, Y2, Z2), C (X3, Y3, Z3), and D (X4, Y4, Z4).

[0065] The image coordinate S3 is placed in a plane located at the focal length from the origin on the Z axis of the camera coordinate S2. The origin O3 of the image coordinate S3 is placed in the upper left corner. The image coordinate S3 is set within a range of w in the width direction from the origin O3 and h in the downward direction from the origin O3. The positions of the four corners of the image coordinate S3 are (0,0), (w,0), (w,h), and (0,h). The intersection of the optical axis and the plane containing the image coordinate S3 is represented by (w / 2, h / 2) in the image coordinate S3.

[0066] In the image coordinate S3, the image of the reference code C1 is the area surrounded by the intersection of a straight line connecting the origin O2 of the camera coordinate S2 and the center points A to D of the four position detection patterns of the reference code C1 with a plane including the image coordinate S3.

[0067] In the image coordinate S3, the central points of the four position detection patterns of the image of the reference code C1 are represented by a(x1, y1), b(x2, y2), c(x3, y3), and d(x4, y4).

[0068] In the camera coordinate system S2, vector Sa pointing from the origin O2 to point a (x1, y1), vector Sb pointing from the origin O2 to point b (x2, y2), vector Sc pointing from the origin O2 to point c (x3, y3), and vector Sd pointing from the origin O2 to point d (x4, y4) are expressed by the following formula 1. Here, z represents the distance from the origin O2 to the focal point of the lens.

[0069]

number

[0070] Here, each of the central points of the four position detection patterns of reference code C1 is located on an extension of a straight line connecting the central points of the four position detection patterns of the image of reference code C1 to origin O2. The vectors pointing from origin O2 to central points A to D of the four position detection patterns of reference code C1 are expressed by the following formula 2.

[0071]

number

[0072] Here, k, l, m, and n are real numbers equal to or greater than 1. In addition, since the reference code C1 is a square, the following formula 3 holds.

[0073]

number

[0074] Here, if the horizontal dimension of the reference code C1 is W and the vertical dimension is H, the following formula 4 holds.

[0075]

number

[0076] The calculation unit 232 solves this simultaneous equation to obtain the positions of the center points A, B, C, and D of the four position detection patterns of the reference code C1 expressed in the camera coordinate system S2.

[0077] Next, the calculation unit 232 transforms the coordinates expressed in the camera coordinates S2 into the reference coordinates S1. An affine transformation including translation and rotation of three-dimensional coordinates can be expressed by a matrix product.

[0078] Here, if the reference code C1 represented in the camera coordinate S2 is the origin (tx, ty, tz), the vector pointing along the X axis of the reference coordinate S1 represented in the camera coordinate S2 is (Xx, Xy, Xz), the vector pointing along the Y axis of the reference coordinate S1 is (Yx, Yy, Yz), and the vector pointing along the Z axis of the reference coordinate S1 is (Zx, Zy, Zz), then the matrix M1 that converts the coordinates represented in the camera coordinate S2 to the reference coordinate S1 is expressed by the following equation 5.

[0079]

number

[0080] With the reference code C1 as the origin, for example, the position of the center of the square can be used. The direction from the midpoint of the line segment CD to the midpoint of the line segment AB can be used as the Y-axis vector of the reference coordinate S1. The direction of the cross product of vectors AB and AD can be used as the Z-axis vector of the reference coordinate S1. Note that the origin of the reference code C1 may be a position other than the center of the square. The direction of the cross product of vectors AB and AD is the orientation of the reference code C1.

[0081] By using the above formula (5), the position of the origin O2 expressed in the camera coordinate S2 is converted to the position of the reference coordinate S1, so that the position of the lens of the camera 13 can be expressed in the reference coordinate S1. In this embodiment, the position of the terminal 10 is expressed by the position of the lens of the camera 13. In this way, the calculation unit 232 calculates the position of the terminal 10 expressed in the reference coordinate S1. Note that the position of the terminal 10 may be expressed by applying a predetermined offset to the position of the lens of the camera 13.

[0082] When the position of the terminal 10 represented by the reference coordinates S1 is calculated, the control unit 231 may display on the user IF 12 that the position of the terminal 10 has been calculated, to notify the user.

[0083] Next, the process of calculating the position of the article 60 in step S107 described above will be described below.

[0084] After acquiring the image of the reference code C1, the user moves to the article 60 whose position is to be registered. The terminal 10 also moves together with the user. The calculation unit 232 obtains the amount of movement from the position of the terminal 10 when the position of the terminal 10 in the reference coordinates S1 is calculated to the article 60 by integrating the position information detected by the first sensor 14. The calculation unit 232 also obtains the amount of change in orientation from the position of the terminal 10 when the position of the terminal 10 in the reference coordinates S1 is calculated to the position where the image of the article 60 is acquired by integrating the angle information detected by the second sensor 15.

[0085] Such a technology for estimating the self-position using the amount of movement and the amount of change in orientation of the terminal 10 is known as PDR (Pedestrian Dead Reckoning). It is also realized as an IMU (Inertial Measurement Unit) combining sensors such as an acceleration sensor and a gyro sensor. Here, it is preferable from the viewpoint of obtaining accurate values ​​of the amount of movement and the amount of change in orientation of the terminal 10 to correct the amount of movement and the amount of change in orientation using an image acquired by the camera 13 while the terminal 10 is moving. Such a technology is known as VIO (Visual-Inertial Odometry). Software or modules using VIO include AR frameworks, such as ARkit (registered trademark). The AR framework has various functions for developing an AR application. Here, it is assumed that the AR framework has a function of displaying an image captured by a camera on the screen of the terminal by perspective transformation and a tracking (movement distance detection) function using various sensors (LiDAR, gyroscope, linear accelerometer, etc.) and a camera.

[0086] Here, a matrix Mw that converts a position expressed by coordinates SA (see FIG. 1) with the position of camera 13 at the time of starting camera 13 as the origin into a position of reference coordinates S1 is expressed by the following formula 6.

[0087]

number

[0088] Matrix M2 converts a position expressed by coordinate SA (see FIG. 1) with the position of camera 13 at the time of starting camera 13 as the origin, into the position of camera coordinate S2 of camera 13 of terminal 10 located at any position within warehouse 50. Coordinate SA is set with the camera position of camera 13 at the time of starting camera 13 as the origin, and the camera direction as the azimuth. The any position within warehouse 50 is the position of camera 13 when the image of reference code C1 was captured.

[0089] Usually, the camera 13 is activated before acquiring an image of the reference code C1. The matrix M2 is calculated based on the position of the terminal 10 and the orientation of the camera 13 at the time when the camera 13 is activated as the terminal 10 moves, and the amount of movement and orientation change of the terminal 10 thereafter. The matrix M2 can be calculated in the same manner as in the above formula 5. For example, an AR framework, specifically ARkit (registered trademark), etc. can be used to calculate the matrix M2.

[0090] The user uses the camera 13 of the terminal 10 to capture an image showing the vicinity of the article 60. On the screen of the user IF 12 on which the image of the article 60 is displayed, the user taps a point in the area in which the article 60 is displayed. The image showing the vicinity of the article 60 preferably includes the article 60.

[0091] The calculation unit 232 detects the distance to the article 60 in the direction of the tapped position in the image coordinates S3 using the third sensor 16. The distance between the tapped position in the image coordinates S3 and the camera 13 is an example of the distance between the camera 13 and the vicinity of the article.

[0092] A method in which the calculation unit 232 calculates the position of the article 60 based on the tapped position will be described below. In the camera coordinates S2, a vector Sa connecting the origin O2 and the tapped position on the image coordinates S3 is expressed, for example, by the following formula 7. Here, it is assumed that the user taps a certain point a on the image of the article 60.

[0093]

number

[0094] Since the article 60 is on the extension of the vector Sa that points from the origin O2 to the point a(x1, y1) in the camera coordinate system S2, the following formula 8 holds: The vector SA points from the origin O2 to a point on the article 60 that corresponds to the tapped point.

[0095]

number

[0096] Here, k is a real number equal to or greater than 1. If the distance to the article 60 in the direction of the tapped position in the image coordinate S3 is L, then the following formula 9 holds. L in the following formula 9 is detected using the third sensor 16 (distance measuring unit).

[0097]

number

[0098] The calculation unit 232 solves this simultaneous equation to determine the position of a point C1 on the article 60 expressed in the camera coordinates S2.

[0099] Then, the calculation unit 232 converts the position C1 expressed in the camera coordinates S2 into a position in the reference coordinates S1.

[0100] Matrix Mcw that converts position C1 expressed in camera coordinates S2 into a position in reference coordinates S1 is expressed by the following formula 10. Here, matrix M2 converts a position expressed in coordinates SA, which has the origin at the position of camera 13 when camera 13 is started, into a position in camera coordinates S2 of camera 13 of terminal 10 located at an arbitrary position in warehouse 50. The arbitrary position in warehouse 50 is a position where an image of item 60 is captured by camera 13.

[0101]

number

[0102] The calculation unit 232 uses Mcw expressed by the above formula 10 to convert the position C1 expressed in the camera coordinates S2 into the position B1 in the reference coordinates S1, as expressed by the following formula 11.

[0103]

number

[0104] The control unit 231 transmits the position B1 of the item 60 together with the item ID and the registration information for identifying the item 60 to the server 30. The item ID and the registration information are input or selected from a predetermined list by the user operating the user IF 12.

[0105] The registration information registered in the registration information column 203 of the item management table 200 is a one-point tap because one point in the area showing the item 60 has been tapped. In the one-point tap, as described above, the calculation unit 232 calculates the position of the item shown in the reference coordinates S1 based on the one point selected on the image of the item 60 shown on the screen of the user IF 12.

[0106] The calculation unit 232 also calculates the orientation of the article 60. The calculation unit 232 calculates a plane including the corresponding position of the article 60 corresponding to the tapped position in the image coordinates S3. For example, the calculation unit 232 uses the third sensor 16 to scan the distance between the terminal 10 and an area including the corresponding position of the article 60, and generates a polygon model representing the shape of the article 60. The calculation unit 232 selects three or more vertices that surround the corresponding position and have a constant distance from the terminal 10. The calculation unit 232 determines the normal direction of the plane defined by the positions of these vertices expressed in the reference coordinates S1 as the orientation of the article 60.

[0107] The position of the article 60 is registered as a registered position pos, and the orientation of the article 60 is registered as a registered orientation rot in the article management table 200.

[0108] This allows the user to select the target item 60 without placing a burden on the user.

[0109] It should be noted that the method of registering the location of the article may be other than the above. Next, the other registration method will be described below.

[0110] (4-point tap) The calculation unit 232 may calculate, as the position of the article 60, a plane represented by reference coordinates S1 defined based on four points selected on an image representing the vicinity of the article 60 displayed on the screen of the user IF 12. The distance between the camera 13 and the four points selected on the image representing the vicinity of the article 60 is an example of the distance between the camera 13 and the vicinity of the article.

[0111] This reduces the burden on the user while enabling the acquisition of object posture information that cannot be calculated with a single tap.

[0112] On the screen of the user IF 12 on which an image of the article 60 is displayed, the user taps four points in the area in which the article 60 is displayed.

[0113] The calculation unit 232 uses the third sensor 16 to detect the distance between the camera 13 and the article 60 located in the direction of the four tapped points in the image coordinate S3.

[0114] The calculation unit 232 obtains four positions B1 to B4 of the reference coordinate S1 based on the four tapped points on the image in the same manner as described above. The plane represented by the reference coordinate S1 is defined by the four positions B1 to B4. The position of the article 60 is represented by this plane.

[0115] The control unit 231 transmits the positions B1 to B4 of the item 60 together with the item ID for identifying the item 60 and the registration information to the server 30. The registration information is four-point tap.

[0116] The calculation unit 232 may calculate, as the position of the article 60, a plane represented by a reference coordinate S1 defined based on at least three points selected on the image of the article 60 displayed on the screen of the user interface 12.

[0117] This makes it possible to obtain a planar image of an item with a minimum of three points, and by using the number of taps to obtain the outline of the item, the image can be cut out and associated with the item and registered in the item management table 200 in Figure 4(B), thus improving the convenience of management.

[0118] The calculation unit 232 may generate a distance image using the third sensor 16 for the area on the image coordinates S3 where the item 60 is displayed, select an area where the distance from the terminal 10 is within a predetermined reference value range, and set the plane representing this area as the position of the item 60. In this case, the calculation unit 232 may also select four points on the plane representing this area, and transmit the positions of these four points to the server 30 as the position of the item 60. In addition, the normal direction of a point on the plane may be transmitted to the server 30 as an orientation, and registered in the item management table 200 in association with the item, and this may be set as the orientation of the item 60.

[0119] (First Code) 6(A) is a diagram for explaining an example of a process for registering the position of an item 60. An item code D1 is placed on the item 60. The item code D1 is an example of a first item code.

[0120] The calculation unit 232 calculates a plane containing the item code D1 represented by the reference coordinates S1 based on the image of the item code D1 shown in the image acquired by the camera 13, the distance between the camera 13 and the item code D1, and the current position and current orientation of the terminal 10 calculated by the calculation unit 232, and determines that a specified area within the plane containing the item code D1 is the position of the item 60 represented by the reference coordinates S1.

[0121] The camera 13 of the terminal 10 is operated by a user to capture an image showing the vicinity of the item 60 including the item code D1. The item code D1 is a two-dimensional code having a rectangular outer shape. Position detection patterns are arranged at the four corners of the item code D1.

[0122] The calculation unit 232 analyzes the image of the item code D1 to obtain a code ID contained in the code, and based on this information, determines that the item code D1 is a code for calculating the position of the item 60. The calculation unit 232 detects the distance between the camera 13 and the item code D1 in the direction of the center points of the four position detection patterns of the item code D1 in the image coordinate S3, using the third sensor 16.

[0123] The calculation unit 232 calculates the position of the center point of each of the four position detection patterns of the item code D1 represented by the camera coordinate S2, and then calculates the position of the center point of each of the four position detection patterns of the item code D1 represented by the reference coordinate S1. The positions of the center points of each of the four position detection patterns of the item code D1 define a plane including the item code D1. The calculation unit 232 determines a predetermined area in the plane represented by the center points of each of the four position detection patterns of the item code D1 as the position of the item 60 represented by the reference coordinate S1. The predetermined area in the plane may be, for example, an area surrounded by the center points of each of the four position detection patterns of the item code D1.

[0124] Furthermore, the calculation unit 232 calculates the normal direction of the plane represented by the reference coordinates S1 as the orientation of the item 60. The normal direction of the plane represented by the reference coordinates S1 represents the orientation of the item code D1. The orientation of the item code D1 is calculated in the same manner as the orientation of the reference code C1. The position of the item 60 is registered in the item management table 200 as the registered position pоs and the orientation of the item 60 is registered as the registered orientation rоt.

[0125] The control unit 231 transmits the positions of the center points of each of the four position detection patterns of the item code D1, together with the orientation of the item, the item ID for identifying the item 60, the registration information, and the specifications of the item code D1 to the server 30. The registration information is a first code. The specifications of the item code D1 are input or selected from a predetermined list by the user operating the user IF 12.

[0126] As a result, when a two-dimensional code is used to register the position of the article 60, highly accurate position information and orientation can be calculated without the user having to perform an action such as tapping.

[0127] (Second Code) 6(B) is a diagram for explaining another example of the process of registering the position of the item 60. The item 60 is placed on the shelf 51. An item code D2 is displayed on the shelf 51 near the item 60. The item code D2 is displayed on the shelf 51 with its relative positional relationship with the item 60 determined. The item code D2 is an example of a second item code.

[0128] This method eliminates the need to attach a code to an item, and also offers the benefits of using a two-dimensional code even when there is no space to attach a code to an item.

[0129] The calculation unit 232 calculates the position of the item code D2 represented by the reference coordinates S1 based on the image of the item code D2 represented in the image representing the vicinity of the item 60 acquired by the camera 13, the distance between the camera 13 and the item code D2, and the current position and current orientation of the terminal 10 calculated by the calculation unit 232, and calculates the position of the item 60 represented by the reference coordinates S1 based on the position of the item code D2 and the positional relationship between the item code D2 and the item 60. The image representing the vicinity of the item 60 preferably includes the item code D2. The image representing the vicinity of the item 60 may or may not include the item 60.

[0130] The camera 13 of the terminal 10 is operated by a user to capture an image including the item code D2. The item code D2 is a two-dimensional code having a rectangular shape. The calculation unit 232 analyzes the image of the item code D2 to obtain information contained in the code, and determines that the item code D2 is a code for calculating the position of the item 60 based on this information.

[0131] The calculation unit 232 uses the third sensor 16 to detect the distance between the camera 13 and the article code D2 located in the direction of the center points of the four position detection patterns of the article code D2 in the image coordinate S3.

[0132] The calculation unit 232 calculates the position of the center point of each of the four position detection patterns of the item code D2 represented by the camera coordinate S2, and then calculates the position of the item 60. There are various possible methods for calculating the position of the item 60 corresponding to the item code D2, but the calculation unit 232 calculates the position of the item 60 represented by the reference coordinate S1, assuming that the item code D2 is affixed at a position relative to the item 60.

[0133] The calculation unit 232 calculates the position of the item 60 represented by the reference coordinates S1 based on the positions of the center points of the four position detection patterns of the item code D2 represented by the reference coordinates S1 and the positional relationship between the item code D2 and the item 60. The positional relationship between the item code D2 and the item 60 is acquired, for example, by analyzing the item code D2.

[0134] For example, if the positions of the center points of the four position detection patterns of the item code D2 represented by the reference coordinate S1 are A (X1, Y1, Z1), B (X2, Y2, Z2), C (X3, Y3, Z3), and D (X4, Y4, Z4), the relative coordinates A' (X1 + ax, Y1 + ay, Z1), B' (X2 + bx, Y2 + by, Z2), C' (X3 + cx, Y3 + cy, Z3), and D' (X4 + dx, Y4 + dy, Z4) are obtained according to the positional relationship between the item code D2 and the item 60. The plane consisting of these four points A' B' C' D' is the position of the item 60. These relative coordinates A' B' C' D' may be stored in the item management table 200 in association with the item code D2.

[0135] The control unit 231 transmits the positions of the four points representing the item 60, together with an item ID for identifying the item 60, registration information, and specifications of the item code D2 to the server 30. The registration information is a second code. The specifications of the item code D2 are input or selected from a predetermined list by the user operating the user IF 12.

[0136] This registration method is effective when it is not possible to affix the item code D1 to the item 60.

[0137] (Features) The camera 13 of the terminal 10 is operated by the user to capture an image in which the item 60 is displayed at the center of the display screen of the user IF 12. The user taps a point in the center of the image including the item 60. The calculation unit 232 calculates feature points and feature amounts (e.g., SIFT, SURF, ORB) from a predetermined area in the image centered on the tapped point. The feature points and feature amounts extracted from the image representing the item 60 are registered in the item management table 200 as registered feature amounts SP of the item 60.

[0138] The calculation unit 232 detects the distance between the camera 13 and the article 60 located in the direction of the tapped position in the image coordinates S3, using the third sensor 16. The distance between the camera 13 and the article 60 located in the direction of the tapped position in the image coordinates S3 is an example of the distance between the camera 13 and the vicinity of the article 60.

[0139] The calculation unit 232 calculates the position of the article 60 expressed in the reference coordinates S1 based on the distance between the article 60 and the camera 13 and the current position and current orientation of the terminal 10 calculated by the calculation unit 232.

[0140] Furthermore, the calculation unit 232 calculates the orientation of the article 60. First, the calculation unit 232 calculates a plane including the corresponding position of the article 60 corresponding to the tapped position in the image coordinates S3. For example, the calculation unit 232 uses the third sensor 16 to scan the distance between the terminal 10 and an area including the corresponding position of the article 60, and generates a polygon model representing the shape of the article 60. The calculation unit 232 selects three or more vertices that surround the corresponding position and have a constant distance from the terminal 10. The calculation unit 232 determines the normal direction of the plane defined by the positions of these vertices expressed in the reference coordinates S1 as the orientation of the article 60.

[0141] The position of the article 60 is registered as a registered position pos, and the orientation of the article 60 is registered as a registered orientation rot in the article management table 200.

[0142] Next, the item search processing of the item management system 1 will be described below with reference to Fig. 7 to Fig. 12. Fig. 7 is a hardware configuration diagram of the terminal 100 that searches for the item 60. Fig. 8 is a sequence diagram of the item search processing of the item management system 1. Figs. 9 to 12 are examples of operational flowcharts of the search processing of the terminal 100 according to the first to fourth embodiments.

[0143] The terminal 100 is used to search for the item 60. The terminal 100 has the same hardware configuration as the above-mentioned terminal 10. The item management system 1 may include multiple terminals 100, but since each terminal has the same configuration and executes the same processing with respect to item management processing, the following description will be given of one terminal 100. The terminal 100 is an example of an item search device.

[0144] The terminal 100 includes a communication unit 111, a user interface (IF) 112, a camera 113, a first sensor 114, a second sensor 115, a third sensor 116, an item management unit 117, and the like.

[0145] The item management unit 117 includes a control unit 1231 , a calculation unit 1232 , a determination unit 1233 , and a generation unit 1234 .

[0146] The communication unit 111 corresponds to the communication unit 11, the IF 112 corresponds to the IF 12, the camera 113 corresponds to the camera 13, the first sensor 114 corresponds to the first sensor 14, the second sensor 115 corresponds to the second sensor 15, the third sensor 116 corresponds to the third sensor 16, and the item management unit 117 corresponds to the item management unit 17. The control unit 1231 corresponds to the control unit 231, and the calculation unit 1232 corresponds to the calculation unit 232.

[0147] Here, IF 112 is an example of a display unit, camera 113 is an example of an image acquisition unit, first sensor 114 is an example of a first sensor, and second sensor 15 is an example of a second sensor.

[0148] The above-mentioned explanations of the communication unit 11, user interface (IF) 12, camera 13, first sensor 14, second sensor 15, third sensor 16, and item management unit 17 of the terminal 10 are appropriately applied to the communication unit 111, user interface (IF) 112, camera 113, first sensor 114, second sensor 115, third sensor 116, and item management unit 117 of the terminal 100. In addition, the above-mentioned explanations of the control unit 231 and calculation unit 232 of the item management unit 17 of the terminal 10 are appropriately applied to the control unit 1231 and calculation unit 1232 of the item management unit 117.

[0149] The user uses the terminal 100 to search for an article registered in the article management table 200 stored in the storage device 32 of the server 30 .

[0150] The processes from steps S201 to S204 are similar to the processes from steps S101 to S104 described above.

[0151] After the position and orientation of the terminal 100 expressed by the reference coordinates S1 are calculated (step S204), the control unit 1231 transmits a search request for information on the item to be searched for to the server 30 via the macrocell base station 41 and the communication network 40 (step S205). The search request preferably includes a code ID representing the reference code C1.

[0152] Next, in response to the search request, the search unit 343 of the server 30 refers to the item management table 200 stored in the storage device 32 and generates search information including a list of items whose positions are registered at the reference coordinates S1. The search information includes information such as item IDs, names, and states. The search unit 343 transmits the search information to the terminal 100 via the communication network 40 and the macrocell base station 41 (step S206).

[0153] Next, the control unit 1231 of the terminal 100 displays the search information on the user IF 112. The user selects an item to be searched for by operating the user IF 112. The control unit 1231 transmits a location request for the location of the selected item to the server 30 via the macrocell base station 41 and the communication network 40 (step S207).

[0154] Next, in response to the location request, the search unit 343 of the server 30 refers to the item management table 200 stored in the storage device 32, and transmits location information and registration information indicating the location of the item to be searched for to the terminal 100 via the communication network 40 and the macrocell base station 41 (step S208). The location information may include a registered location and a registered direction.

[0155] Next, the user moves to the item 60 to be searched for together with the terminal 100 with the camera 113 activated (step S209). The environment inside the warehouse 50 captured by the camera 113 is displayed on the user IF 112. The generation unit 1234 calculates the direction from the position of the terminal 100 to the item 60 based on the position and orientation of the terminal 100 expressed in the reference coordinate system S1 and the position of the item 60 to be searched for. The generation unit 1234 generates a display that guides the user IF 112 to the position of the item 60. The control unit 1231 may display the display generated by the generation unit 1234 superimposed on the environment inside the warehouse 50.

[0156] Next, the terminal 100 executes a search process to search for the article 60 (step S210). The terminal 100 generates a search result in response to the search process. This search process will be described later with reference to Figs. 9 to 12.

[0157] Next, the control unit 1231 of the terminal 100 transmits the search result to the server 30 via the macrocell base station 41 and the communication network 40 (step S211).

[0158] Next, the searching unit 343 of the server 30 updates the item management table 200 stored in the storage device 32 according to the search result (step S213), and ends the series of processes. The searching unit 343 changes the information in the status column 206 associated with the discovered item in the item management table 200 stored in the storage device 32 from "unsearched" to "searched" according to the search result.

[0159] Next, first to fourth embodiments of the search process in the above-mentioned step S210 will be described below with reference to FIGS.

[0160] 9 is an example of an operation flowchart of a first embodiment of a search process of the terminal 100. In the search process of this embodiment, the search process described below can be executed regardless of the registration information registered in the registration information column 203 of the item management table 200.

[0161] First, the determination unit 1233 determines whether the registered position of the item 60 represented by the reference coordinates S1 registered in the item management table 200 is included in the field of view of the camera 113 (step S301). The determination unit 1233 determines whether the registered position of the item 60 to be searched is included in the field of view of the camera 113 based on internal parameters including the focal length and field of view of the camera 113, external parameters including the mounting position of the camera 113 on the terminal 100, the current position and current orientation of the terminal 100, and the registered position of the item 60. The determination unit 1233 is an example of a first determination unit. Here, determining whether the registered position of the item 60 is included in the field of view of the camera 113 includes comparing the current position of the terminal 100 with the position of the item 60 to be searched and determining that the positions are close to each other. In addition, the direction to the position of the item 60 to be searched may be obtained with the current position of the terminal 100 as the origin, and the direction may be compared with the current orientation of the terminal 100 to determine whether the position is within a certain range. The current orientation of the terminal 100 is an orientation based on a normal vector to the plane of the reference code C1. Specifically, this orientation can be expressed by Euler angles. For example, the determination unit 1233 obtains the distance between the registered position of the searched item 60 represented by the reference coordinates S1 and the current position of the terminal 100 represented by the reference coordinates S1, and determines whether this distance is equal to or smaller than a predetermined reference distance. In addition, when the current orientation of the terminal 100 is vector a and the orientation of the item 60 as seen from the terminal 100 is vector b, the determination unit 1233 determines whether the orientation is within a certain range based on the angle θ=arccos(a·b / |a||b|). Here, a·b is the inner product of vector a and vector b, |a| is the magnitude of vector a, and |b| is the magnitude of vector b.

[0162] If the registered position of the article 60 is included in the field of view of the camera 113 (step S301-Yes), the control unit 1231 acquires an image of the environment included in the field of view of the camera 113 using the camera 113 of the terminal 100 (step S302). On the other hand, if the registered position of the article 60 is not included in the field of view of the camera 113 (step S301-No), the process returns to before step S301. It is considered that the registered position of the article 60 will be included in the field of view of the camera 113 as the user approaches the article 60 to be searched for together with the terminal 100.

[0163] Next, the control unit 1231 uses the third sensor 116 to measure a distance L1 between the terminal 100 and an object located in the direction of the registered position of the article 60 to be searched for relative to the terminal 100 (step S303). Note that the order of steps S302 and S303 may be reversed.

[0164] Next, the determination unit 1233 determines whether the distance L1 matches the distance L2 between the registered position of the article 60 to be searched and the position of the terminal 100 (step S304). The determination unit 1233 calculates the distance L2 based on the registered position of the article 60 represented by the reference coordinates S1 and the position of the terminal 100. If the difference between the distance L1 and the distance L2 is equal to or less than a predetermined reference distance R1, the determination unit 1233 determines that the distance L1 matches the distance L2. On the other hand, if the difference between the distance L1 and the distance L2 exceeds the predetermined reference distance R1, the determination unit 1233 determines that the distance L1 does not match the distance L2. The determination unit 1233 is an example of a second determination unit.

[0165] If the distance L1 and the distance L2 match (Yes in step S304), the determination unit 1233 determines that the registered position of the searched item 60 matches the current position of the item (step S305).

[0166] Next, the generation unit 1234 arranges the item information about the item 60 at a position determined based on the registered position of the item 60 to be searched for in the image acquired by the camera 113, and generates an extended image to be displayed on the UI 112 (step S306). The UI 112 is an example of a display unit.

[0167] Next, the generating unit 1234 generates an extended image in which the item information is displayed so as to overlap with the item 60 to be searched for (step S307). The generating unit 1234 may generate the item information based on the registration information. For example, when the registration information is one-point tap, the generating unit 1234 may generate item information having a spherical figure, and when the registration information is four-point tap, the generating unit 1234 may generate item information having a planar figure. Note that other expressions may be used as the item information.

[0168] The generation unit 1234 places the item information at the registration position of the item 60 to be searched for, which is represented by the reference coordinates S1. Next, the generation unit 1234 projects the item information onto a plane represented by the image coordinates S3 using viewpoint projection transformation. Then, an extended image is generated in which the item information is represented so as to overlap with the item 60 to be searched for.

[0169] Next, the control unit 1231 displays the extended image generated by the generation unit 1234 on the user IF 112 (step S308). The extended image showing the item information is displayed so as to overlap with the item 60 to be searched for on the user IF 112. This allows the user to recognize that the item 60 to be searched for has been found. The user operates the user IF 112 to input discovery information indicating that the item 60 to be searched for has been found to the terminal 100.

[0170] Next, the control unit 1231 generates a search result including the discovery information, and ends the series of processes (step S309). The search result includes, for example, information indicating that the searched item 60 has been found, an item ID, and the like.

[0171] On the other hand, if the distance L1 and the distance L2 do not match (step S304-No), the determination unit 1233 determines that the registered position of the searched item 60 does not match the current position of the item (step S310).

[0172] The determination unit 1233 determines whether the distance L1 is shorter than the distance L2 (step S311). If the distance L1 exceeds the reference distance R1 and is shorter than the distance L2, the determination unit 1233 determines that the distance L1 is shorter than the distance L2. On the other hand, if the distance L1 exceeds the reference distance R1 and is longer than the distance L2, the determination unit 1233 determines that the distance L1 is longer than the distance L2.

[0173] If the distance L1 is shorter than the distance L2 (step S311-Yes), the determination unit 1233 decides to display information indicating that an object other than the item 60 is located between the searched item 60 and the terminal 100, as information indicating that the registered position of the item 60 does not match the current position of the item 60, on the user IF 112 (step S312). If an object other than the item 60 is located between the searched item 60 and the terminal 100, it is considered that the item 60 is not shown in the image acquired by the camera 113.

[0174] Next, the generation unit 1234 overlays occlusion information indicating that an object other than the item 60 is located between the searched item 60 and the terminal 100 on the image acquired by the camera 113, thereby generating an extended image (step S307).

[0175] For example, when the registration information is a one-point tap, information having a semi-transparent spherical figure may be generated as the occlusion information. When the registration information is a four-point tap, occlusion information having a semi-transparent planar figure may be generated. Note that other expressions may be used as the occlusion information.

[0176] Next, the control unit 1231 displays the extended image generated by the generation unit 1234 on the user IF 112 (step S308). The extended image in which the occlusion information is superimposed on the image acquired by the camera 113 is displayed on the user IF 112. This allows the user to recognize that an object other than the article 60 is located between the article 60 to be searched and the terminal 100.

[0177] The user can find the searched item 60, for example, by moving another object located between the searched item 60 and the terminal 100. The user operates the user IF 112 to input discovery information indicating that the searched item 60 has been found to the terminal 100.

[0178] Next, the control unit 1231 generates a search result including the discovery information, and ends the series of processes (step S309).

[0179] Furthermore, if the distance L1 is longer than the distance L2 (step S311-No), the determination unit 1233 determines to display information indicating that the searched item 60 is not present on the user IF 112 as information indicating that the registered position of the item 60 does not match the current position of the item 60 (step S313). Since the searched item 60 is not present, it is considered that the item 60 is not shown in the image acquired by the camera 113.

[0180] Next, the generation unit 1234 generates an extended image by superimposing information indicating that the article 60 to be searched is not present on the image acquired by the camera 113 (step S307).

[0181] Next, the control unit 1231 displays the extended image generated by the generation unit 1234 on the user IF 112 (step S308). On the user IF 112, an extended image in which information indicating that there is no article 60 to be searched for is superimposed on the image acquired by the camera 113 is displayed. This allows the user to recognize that there is no article 60 to be searched for.

[0182] The user operates the user IF 112 to input to the terminal 100 discovery information indicating that the searched item 60 was not found.

[0183] Next, the control unit 1231 generates a search result including the discovery information, and ends a series of processes (step S309). This concludes the description of the first embodiment of the search process of the terminal 100.

[0184] 10 is an example of an operation flowchart of a second embodiment of the search process of the terminal 100. In the search process of this embodiment, when the registration information registered in the registration information column 203 of the item management table 200 is a first code, the search process described below can be executed. Note that in this embodiment, the item 60 to be searched is located at a registration position registered in the item management table 200, or is located at a position different from the registration position such that when this registration position is included in the field of view of the camera 113, the item 60 is included in the field of view.

[0185] First, the determination unit 1233 determines whether or not the registered position of the article 60, which is represented by the reference coordinates S1 registered in the article management table 200, is included in the field of view of the camera 113 (step S401).

[0186] If the registered position of the item 60 is included in the field of view of the camera 113 (step S401-Yes), the control unit 1231 acquires an image of the environment included in the field of view of the camera 113 using the camera 113 of the terminal 100 (step S402). It is considered that the image acquired by the camera 113 shows the vicinity of the item 60 to be searched. On the other hand, if the registered position of the item 60 is not included in the field of view of the camera 113 (step S401-No), the process returns to before step S401. It is considered that the registered position of the item 60 will be included in the field of view of the camera 113 as the user approaches the item 60 to be searched together with the terminal 100.

[0187] Next, the calculation unit 1232 calculates the current position of the item 60 represented by the reference coordinates S1 based on the image of the item code D1 shown in the image showing the vicinity of the item 60 acquired by the camera 113, the distance between the terminal 10 (camera 113) and the item code D1, and the current position of the terminal 10 represented by the reference coordinates S1 (step S403). The current position of the item 60 represented by the reference coordinates S1 is calculated in the same way as when the registered position of the item 60 is obtained. The calculation unit 1232 is an example of a first calculation unit.

[0188] Furthermore, the calculation unit 1232 calculates the current orientation of the article 60 represented by the reference coordinates S1 in the same manner as when the registered orientation of the article 60 was calculated.

[0189] The determination unit 1233 determines whether the registered position of the item 60 matches the current position of the item 60 (step S404). For example, when the distance between the registered position of the item 60 and the current position of the item 60 is equal to or less than a predetermined reference distance R2, the determination unit 1233 determines that the registered position of the item 60 matches the current position of the item 60. On the other hand, when the distance between the registered position of the item 60 and the current position of the item 60 exceeds the reference distance R2, the determination unit 1233 determines that the registered position of the item 60 does not match the current position of the item 60.

[0190] The determination unit 1233 may also determine whether the registered orientation of the item 60 matches the current orientation of the item 60. The determination unit 1233 determines whether each of the three Euler angles of the registered orientation of the item 60 matches each of the three Euler angles of the current orientation of the item 60. If the difference between each of the three Euler angles of the registered orientation of the item 60 and each of the three Euler angles of the current orientation of the item 60 is equal to or less than the reference angle, the determination unit 1233 determines that the registered orientation of the item 60 matches the current orientation of the item 60. On the other hand, if any of the differences between each of the three Euler angles of the registered orientation of the item 60 and each of the three Euler angles of the current orientation of the item 60 exceeds the reference angle, the determination unit 1233 determines that the registered orientation of the item 60 does not match the current orientation of the item 60.

[0191] If the registered position of the item 60 matches the current position of the item 60 (step S404-Yes), the generation unit 1234 places item information about the item at a position determined based on the current position of the item 60 in the image acquired by the camera 113, and generates an extended image (step S405). The image acquired by the camera 113 shows the item 60 to be searched, and its position matches the registered position.

[0192] Here, when the registered position of item 60 matches the current position of item 60 and the registered orientation of item 60 matches the current orientation of item 60, the generation unit 1234 may place item information about the item at a position determined based on the current position of item 60 in the image acquired by camera 113, thereby generating an augmented image.

[0193] Next, the generating unit 1234 generates an extended image in which the article information is expressed so as to overlap with the article 60 to be searched for (step S406). The generating unit 1234 may generate the article information based on the registration information. For example, the generating unit 1234 may generate article information having a spherical figure at the registration position of the article 60. Note that other expressions may be used as the article information.

[0194] The generation unit 1234 places the item information at the registration position of the item 60 to be searched for, which is represented by the reference coordinates S1. Next, the generation unit 1234 projects the item information onto a plane represented by the image coordinates S3 using viewpoint projection transformation. Then, an extended image is generated in which the item information is represented so as to overlap with the item 60 to be searched for.

[0195] Next, the control unit 1231 displays the extended image generated by the generation unit 1234 on the user IF 112 (step S407). The extended image showing the item information is displayed so as to overlap with the item 60 to be searched for on the user IF 112. This allows the user to recognize that the item 60 to be searched for has been found. The user operates the user IF 112 to input discovery information indicating that the item 60 to be searched for has been found to the terminal 100.

[0196] Next, the control unit 1231 generates a search result including the discovery information, and ends the series of processes (step S408). The search result includes, for example, information indicating that the searched item 60 has been found, an item ID, and the like.

[0197] On the other hand, if the registered position of the item 60 does not match the current position of the item 60 (step S404-No), the generating unit 1234 generates an extended image by arranging item information about the item 60 at a position determined based on the current position of the item 60 to be searched for in the image acquired by the camera 113 (step S409). Although the item 60 to be searched for is shown in the image acquired by the camera 113, its position is different from the registered position.

[0198] Next, the generating unit 1234 generates an extended image in which the item information is displayed so as to overlap with the current position of the item 60 to be searched for (step S406). The generating unit 1234 may generate the extended image so that the item information overlaps with the current position of the item 60 to be searched for and the orientation of the item information coincides with the current orientation of the item 60.

[0199] The generating unit 1234 may generate the item information based on the registration information. For example, the generating unit 1234 may generate the item information having a spherical graphic at the current position of the item 60. Note that other expressions may be used as the item information.

[0200] The generating unit 1234 places the item information at the current position of the searched item 60 represented by the reference coordinates S1. Next, the generating unit 1234 projects the item information onto a plane represented by the image coordinates S3 using viewpoint projection transformation. Then, it generates an augmented image in which the item information is represented so as to overlap with the searched item 60.

[0201] Next, the control unit 1231 displays the extended image generated by the generation unit 1234 on the user IF 112 (step S407). The extended image showing the item information is displayed so as to overlap with the item 60 to be searched for on the user IF 112. This allows the user to recognize that the item 60 to be searched for has been found. The user operates the user IF 112 to input discovery information indicating that the item 60 to be searched for has been found to the terminal 100.

[0202] Next, the control unit 1231 generates a search result including the discovery information, and ends the series of processes (step S408). The search result includes, for example, information indicating that the searched item 60 has been found, an item ID, and the like. This concludes the description of the second embodiment of the search process of the terminal 100.

[0203] 11 is an example of an operation flowchart of a third embodiment of a search process of the terminal 100. In the search process of this embodiment, when the registration information registered in the registration information column 203 of the item management table 200 is a feature amount, the search process described below can be executed.

[0204] First, the determination unit 1233 determines whether or not the registered position of the article 60, which is represented by the reference coordinates S1 registered in the article management table 200, is included in the field of view of the camera 113 (step S501).

[0205] If the registered position of the article 60 is included in the field of view of the camera 113 (step S501-Yes), the determination unit 1233 determines whether or not the distance between the registered position of the article 60 to be searched, represented by the reference coordinates S1, and the current position of the terminal 100, represented by the reference coordinates S1, is equal to or less than a predetermined reference distance R3 (step S502). If the registered position of the article 60 is not included in the field of view of the camera 113 (step S501-No), the process returns to before step S501. It is considered that the registered position of the article 60 will be included in the field of view of the camera 113 as the user approaches the article 60 to be searched together with the terminal 100.

[0206] If the distance between the registered position of the article 60 and the current position of the terminal 100 is equal to or less than the reference distance R3 (step S502-Yes), the control unit 1231 uses the camera 113 of the terminal 100 to acquire an image of the environment included in the field of view of the camera 113 (step S503). Here, it is preferable that the camera 113 of the terminal 100 is operated by the user to acquire an image in which the article is displayed in the center of the display screen of the user IF 112. On the other hand, if the distance between the registered position of the article 60 and the current position of the terminal 100 exceeds the reference distance R3 (step S502-No), the process returns to before step S501. The process may also return to before step S502.

[0207] Next, the calculation unit 1232 calculates feature points and feature amounts (e.g., SIFT, SURF, ORB) from a predetermined area in the image centered on a central point of the image acquired by the camera 113 (step S504). It is considered that the article 60 to be searched for is depicted in the center of the image acquired by the camera 113. The calculation unit 1232 calculates the current feature amount of the article 60 depicted in the current image of the article 60 acquired by the camera 113 in the same manner as when the position of the article 60 is registered.

[0208] Since the processing load of calculating feature points and feature amounts from an image is large, in this embodiment, the processing is performed after the distance between the registered position of the article 60 and the current position of the terminal 100 becomes equal to or less than a reference distance R3. The reference distance R3 can be determined depending on the processing load of calculating feature points and feature amounts from an image.

[0209] Next, the calculation unit 1232 determines whether or not the current feature amount of the article 60 shown in the current image of the article 60 acquired by the camera 113 matches the registered feature amount of the article 60 (step S505).

[0210] The calculation unit 1232 performs a matching process between the current feature amount of the article 60 and the registered feature amount of the article 60. As the matching process, for example, a Brute-Force matcher can be used.

[0211] First, the calculation unit 1232 calculates the similarity of each feature point included in the current feature of the article 60 between the feature points of the registered feature of the article 60, detects the most similar feature point, and obtains the pair of feature points and the similarity (distance). The similarity is expressed, for example, from 0.0 (perfect match) to 1.0 (perfect mismatch).

[0212] Then, the calculation unit 1232 selects the top N pairs (N is a predetermined natural number) with the highest similarity from among the pairs of feature points. If the sum of the similarities of the selected N pairs is less than a predetermined reference similarity M (M is a predetermined real number), the determination unit 1233 determines that the current feature amount of the item 60 matches the registered feature amount of the item 60.

[0213] On the other hand, if the sum of the similarities of the selected N sets is equal to or greater than the predetermined reference similarity M, the determination unit 1233 determines that the current feature amount of the article 60 does not match the registered feature amount of the article 60. If the current feature amount of the article 60 does not match the registered feature amount of the article 60, the article 60 to be searched for is not included in the current field of view of the camera 113, and therefore it is considered that there is no article 60 to be searched for.

[0214] If the current feature of item 60 matches the registered feature of item 60 (step S505-Yes), the calculation unit 1232 calculates a position change amount representing information regarding the difference between the registered position of item 60 and the current position of item 60 based on the current feature of item 60 represented in the current image of item 60 acquired by camera 113 and the registered feature of item 60 (step S506).

[0215] Based on the selected N sets of feature points, the calculation unit 1232 calculates a perspective transformation matrix (homography matrix H) that gives a correspondence relationship between the points on the image coordinates S3 at which the current feature amounts of the article 60 are calculated and the points on the image coordinates S3 at which the registered feature amounts of the article 60 are calculated. The homography matrix H can be obtained using a known method such as the least squares method.

[0216] The calculation unit 1232 uses the homography matrix H to find a point P2(xi, yi) in the image in which the registered feature amount of the article 60 is calculated, which corresponds to a point P1(w / 2, h / 2) at the center of the image in which the current feature amount of the article 60 is calculated. Here, the point P2 is expressed as P2(xi, yi)=H·P1(w / 2, h / 2) using the homography matrix H and the point P1(xi, yi).

[0217] The calculation unit 1232 obtains the distance between the points P1 and P2 as the above-mentioned position change amount. This distance is expressed, for example, in terms of the number of pixels.

[0218] Next, the determining unit 1233 determines whether the amount of position change is equal to or smaller than a predetermined reference value (step S507).

[0219] If the amount of position change is equal to or less than the reference value (step S507-Yes), the determination unit 1233 determines that the registered position of the article 60 matches the current position of the article 60 (step S508). The image acquired by the camera 113 shows the article 60 to be searched, and its position matches the registered position.

[0220] Next, the generating unit 1234 places the item information regarding the item 60 at a position determined based on the registered position of the item 60 to be searched for in the image acquired by the camera 113, and generates an extended image (step S509).

[0221] Next, the generating unit 1234 generates an extended image in which the article information is expressed so as to overlap with the article 60 to be searched for (step S510). The generating unit 1234 may generate the article information based on the registration information. For example, the generating unit 1234 may generate article information having a spherical figure at the registration position of the article 60. Note that other expressions may be used as the article information.

[0222] The generation unit 1234 places the item information at the registration position of the searched item 60 represented by the reference coordinates S1. Next, the generation unit 1234 projects the item information onto a plane represented by the image coordinates S3 using viewpoint projection transformation. Then, an extended image is generated in which the item information is represented so as to overlap with the searched item 60.

[0223] Next, the control unit 1231 displays the extended image generated by the generation unit 1234 on the user IF 112 (step S511). The extended image showing the item information is displayed so as to overlap with the item 60 to be searched for on the user IF 112. This allows the user to recognize that the item 60 to be searched for has been found. The user operates the user IF 112 to input discovery information indicating that the item 60 to be searched for has been found to the terminal 100.

[0224] Next, the control unit 1231 generates a search result including the discovery information, and ends the series of processes (step S512). The search result includes, for example, information indicating that the searched item 60 has been found, an item ID, and the like.

[0225] On the other hand, if the registered position of the article 60 does not match the current position of the article 60 (step S507-No), the determination unit 1233 determines that the registered position of the article 60 does not match the current position of the article 60 (step S514). Although the article 60 to be searched is shown in the image acquired by the camera 113, its position is different from the registered position.

[0226] Next, the generation unit 1234 places the item information regarding the item 60 at a position determined based on the current position of the item 60 being searched for in the image acquired by the camera 113, and generates an augmented image (step S515).

[0227] The calculation unit 1232 detects the distance between the camera 113 and the article 60 located in the direction of the center of the image acquired by the camera 113 (the center of the image coordinate S3) using the third sensor 116. The calculation unit 232 calculates the current position of the article 60 represented in the reference coordinate S1 based on the distance between the article 60 and the camera 113 and the current position and current direction of the terminal 10 calculated by the calculation unit 1232.

[0228] Furthermore, the calculation unit 1232 calculates the current orientation of the article 60. First, the calculation unit 1232 calculates a plane including the corresponding position of the article 60 corresponding to the center of the image acquired by the camera 113. For example, the calculation unit 1232 uses the third sensor 116 to scan the distance between the terminal 100 and an area including the corresponding position of the article 60, and generates a polygon model representing the shape of the article 60. The calculation unit 1232 selects three or more vertices that surround the corresponding position and have a constant distance from the terminal 100. The calculation unit 1232 determines the normal direction of the plane defined by the positions of these vertices expressed in the reference coordinates S1 as the current orientation of the article 60.

[0229] Next, the generating unit 1234 generates an extended image in which the item information is displayed so as to overlap with the current position of the item 60 to be searched for (step S510). The generating unit 1234 may generate the extended image so that the item information overlaps with the current position of the item 60 to be searched for and the orientation of the item information coincides with the current orientation of the item 60.

[0230] The generating unit 1234 may generate the item information based on the registration information. For example, the generating unit 1234 may generate the item information having a spherical graphic at the current position of the item 60. Note that other expressions may be used as the item information.

[0231] The generation unit 1234 places the item information at the current position of the searched item 60 represented by the reference coordinates S1. Next, the generation unit 1234 projects the item information onto a plane represented by the image coordinates S3 using viewpoint projection transformation. Then, an augmented image is generated in which the item information is represented so as to overlap with the searched item 60.

[0232] Next, the control unit 1231 displays the extended image generated by the generation unit 1234 on the user IF 112 (step S511). The extended image showing the item information is displayed so as to overlap with the item 60 to be searched for on the user IF 112. This allows the user to recognize that the item 60 to be searched for has been found. The user operates the user IF 112 to input discovery information indicating that the item 60 to be searched for has been found to the terminal 100.

[0233] Next, the control unit 1231 generates a search result including the discovery information, and ends the series of processes (step S512). The search result includes, for example, information indicating that the searched item 60 has been found, an item ID, and the like.

[0234] On the other hand, if the amount of position change exceeds the reference value (step S507-No), the determining unit 1233 determines that the article 60 is not present (step S513).

[0235] Next, the generation unit 1234 generates an extended image by superimposing information indicating that the article 60 to be searched is not present on the image acquired by the camera 113 (step S510).

[0236] Next, the control unit 1231 displays the extended image generated by the generation unit 1234 on the user IF 112 (step S511). On the user IF 112, an extended image in which information indicating that there is no article 60 to be searched for is superimposed on the image acquired by the camera 113 is displayed. This allows the user to recognize that there is no article 60 to be searched for.

[0237] The user operates the user IF 112 to input to the terminal 100 discovery information indicating that the searched item 60 was not found.

[0238] Next, the control unit 1231 generates a search result including the discovery information, and ends a series of processes (step S512). This concludes the description of the third embodiment of the search process of the terminal 100.

[0239] 12 is an example of an operation flowchart of a fourth embodiment of the search process of the terminal 100. In the search process of this embodiment, when the registration information registered in the registration information column 203 of the item management table 200 is a single tap, the search process described below can be executed. Note that in this embodiment, the item 60 to be searched is located at a registration position registered in the item management table 200, or is located at a position different from the registration position such that the item 60 is included in the field of view of the camera 113 when the registration position is included in the field of view of the camera 113.

[0240] First, the determination unit 1233 determines whether or not the registered position of the article 60, which is represented by the reference coordinates S1 registered in the article management table 200, is included in the field of view of the camera 113 (step S601).

[0241] If the registered position of the item 60 is included in the field of view of the camera 113 (step S601-Yes), the control unit 1231 acquires an image of the environment included in the field of view of the camera 113 using the camera 113 of the terminal 100 (step S602). It is considered that the image acquired by the camera 113 shows the item 60 to be searched for. On the other hand, if the registered position of the item 60 is not included in the field of view of the camera 113 (step S601-No), the process returns to before step S601. It is considered that the registered position of the item 60 will be included in the field of view of the camera 113 as the user approaches the item 60 to be searched for together with the terminal 100.

[0242] Next, the calculation unit 1232 calculates the current position and current orientation of the item 60 to be searched, represented by the reference coordinates S1, based on one point selected by the user's tap on the image displayed on the screen of the user IF 112 (step S603). The user taps on an area on the screen of the user IF 112 where the item 60 to be searched is displayed. The current position and current orientation of the item 60 to be searched are calculated in the same manner as the registered position and registered orientation of the item 60 when the registration information is a one-point tap.

[0243] The determination unit 1233 determines whether the registered position of the item 60 matches the current position of the item 60 (step S604). For example, when the distance between the registered position of the item 60 and the current position of the item 60 is equal to or less than a predetermined reference distance R4, the determination unit 1233 determines that the registered position of the item 60 matches the current position of the item 60. On the other hand, when the distance between the registered position of the item 60 and the current position of the item 60 exceeds the reference distance R4, the determination unit 1233 determines that the registered position of the item 60 does not match the current position of the item 60.

[0244] The determination unit 1233 may also determine whether the registered orientation of the item 60 matches the current orientation of the item 60. The determination unit 1233 determines whether each of the three Euler angles of the registered orientation of the item 60 matches each of the three Euler angles of the current orientation of the item 60. If the difference between each of the three Euler angles of the registered orientation of the item 60 and each of the three Euler angles of the current orientation of the item 60 is equal to or less than the reference angle, the determination unit 1233 determines that the registered orientation of the item 60 matches the current orientation of the item 60. On the other hand, if any of the differences between each of the three Euler angles of the registered orientation of the item 60 and each of the three Euler angles of the current orientation of the item 60 exceeds the reference angle, the determination unit 1233 determines that the registered orientation of the item 60 does not match the current orientation of the item 60.

[0245] If the registered position of the item 60 matches the current position of the item 60 (step S604-Yes), the generation unit 1234 places item information about the item at a position determined based on the current position of the item 60 in the image acquired by the camera 113, and generates an extended image (step S605). The image acquired by the camera 113 shows the item 60 to be searched, and its position matches the registered position.

[0246] Here, when the registered position of item 60 matches the current position of item 60 and the registered orientation of item 60 matches the current orientation of item 60, the generation unit 1234 may place item information about the item at a position determined based on the current position of item 60 in the image acquired by camera 113, thereby generating an augmented image.

[0247] Next, the generating unit 1234 generates an extended image in which the article information is expressed so as to overlap with the article 60 to be searched for (step S606). The generating unit 1234 may generate the article information based on the registration information. For example, the generating unit 1234 may generate article information having a spherical figure at the registration position of the article 60. Note that other expressions may be used as the article information.

[0248] The generation unit 1234 places the item information at the registration position of the item 60 to be searched for, which is represented by the reference coordinates S1. Next, the generation unit 1234 projects the item information onto a plane represented by the image coordinates S3 using viewpoint projection transformation. Then, an extended image is generated in which the item information is represented so as to overlap with the item 60 to be searched for.

[0249] Next, the control unit 1231 displays the expanded image generated by the generation unit 1234 on the user IF 112 (step S607). The expanded image showing the item information is displayed so as to overlap with the item 60 to be searched for on the user IF 112. This allows the user to recognize that the item 60 to be searched for has been found. The user operates the user IF 112 to input discovery information indicating that the item 60 to be searched for has been found to the terminal 100.

[0250] Next, the control unit 1231 generates a search result including the discovery information, and ends the series of processes (step S608). The search result includes, for example, information indicating that the searched item 60 has been found, an item ID, and the like.

[0251] On the other hand, if the registered position of the item 60 does not match the current position of the item 60 (step S604-No), the generation unit 1234 places the item information about the item 60 at a position determined based on the current position of the item 60 to be searched for in the image acquired by the camera 113, and generates an extended image (step S609). Although the item 60 to be searched for is shown in the image acquired by the camera 113, its position is different from the registered position.

[0252] Next, the generating unit 1234 generates an extended image in which the item information is displayed so as to overlap with the current position of the item 60 to be searched for (step S606). The generating unit 1234 may generate the extended image so that the item information overlaps with the current position of the item 60 to be searched for and the orientation of the item information coincides with the current orientation of the item 60.

[0253] The generating unit 1234 may generate the item information based on the registration information. For example, the generating unit 1234 may generate the item information having a spherical graphic at the current position of the item 60. Note that other expressions may be used as the item information.

[0254] The generation unit 1234 places the item information at the current position of the searched item 60 represented by the reference coordinates S1. Next, the generation unit 1234 projects the item information onto a plane represented by the image coordinates S3 using viewpoint projection transformation. Then, an augmented image is generated in which the item information is represented so as to overlap with the searched item 60.

[0255] Next, the control unit 1231 displays the expanded image generated by the generation unit 1234 on the user IF 112 (step S607). The expanded image showing the item information is displayed so as to overlap with the item 60 to be searched for on the user IF 112. This allows the user to recognize that the item 60 to be searched for has been found. The user operates the user IF 112 to input discovery information indicating that the item 60 to be searched for has been found to the terminal 100.

[0256] Next, the control unit 1231 generates a search result including the discovery information, and ends the series of processes (step S608). The search result includes, for example, information indicating that the searched item 60 has been found, an item ID, and the like. This concludes the description of the second embodiment of the search process of the terminal 100.

[0257] According to the terminal 10 of the item management system 1 of this embodiment described above, the position of an item is represented by reference coordinates with the reference code displayed in the warehouse as the origin, so that the actual position of an item can be correctly calculated even in a warehouse where GPS information cannot be obtained.

[0258] Furthermore, the terminal 100 of the article management system 1 of this embodiment can search for an article even in a location where GPS information cannot be obtained.

[0259] In the second to fourth embodiments of the search process described above, when the current position of the item 60 differs from the registered position of the item 60, the terminal 100 may correct the position registered in the item management table 200 based on the current position of the item 60. In addition, the terminal 100 may correct its own current position based on the current position of the item 60.

[0260] Furthermore, in the second to fourth embodiments of the search process described above, when the current orientation of the item 60 differs from the registered orientation of the item 60, the terminal 100 may correct the orientation registered in the item management table 200 based on the current orientation of the item 60. Furthermore, the terminal 100 may correct its own current orientation based on the current orientation of the item 60.

[0261] In the above explanation, the positions of all items are expressed with one reference code as the origin. However, if the warehouse is large, the positions of items may be managed using auxiliary coordinates with the origin in the auxiliary code C2, which is determined by the relative positional relationship with the reference code C1.

[0262] In the example shown in Fig. 1, an auxiliary code C2 is displayed on each of the multiple shelves 51. In each shelf 51, the positions of multiple items 60 arranged on this shelf 51 are represented by auxiliary coordinates S4 with the auxiliary code C2 displayed on that shelf 51 as the origin. The positions of the items 60 represented by the auxiliary code S4 are converted to positions in the reference coordinates S1 using predetermined matrices MW2 to MW4. The explanation of the reference coordinates S1 is appropriately applied to the auxiliary coordinates S4.

[0263] 4(A) registers three auxiliary codes C2 displayed on three shelves 51. The three auxiliary codes C2 are identified by different code IDs ref2 to ref4. For example, the calculation unit 232 converts the position of the item represented by the auxiliary coordinates S4 of the auxiliary code C2 of ref2 into the position of the reference coordinates S1 using the matrix MW2.

[0264] The matrix MWn (n is an integer representing a management number that identifies the fixed location code) is expressed by the following formula 12. Here, MW1 is a matrix that converts a position expressed by coordinates SA, whose origin is the camera position of camera 13 when camera 13 is started, into a position of reference coordinates S1. Mw2 is a matrix that converts a position expressed by coordinates SA, whose origin is the camera position of camera 13 when camera 13 is started, into a position expressed in a coordinate system whose origin is auxiliary code C2.

[0265]

number

[0266] In the code management table 100, only the values ​​that vary among the matrix components may be registered in the transformation matrix column 105. In addition, when the matrix MWn does not include transformation components other than rotation and translation, the matrix MWn may be divided into matrices that represent rotation and translation. Here, the matrix that represents rotation may be represented by Euler angles that can be expressed by three components.

[0267] Furthermore, for an item whose position is calculated using auxiliary coordinates S4 with the auxiliary code C2 as the origin, the position of this item expressed in a coordinate system with the auxiliary code C2 as the origin may be registered in the item management table 200. In this case, it is preferable to create a new reference code column in the item management table 200 so that it is possible to identify whether the position of the item is expressed in reference coordinates S1 with the reference code C1 as the reference, or in auxiliary coordinates S4 with the auxiliary code C2 as the reference.

[0268] In the above-described item search process, the user can quickly perform the task of searching for items placed on the shelves 51 by photographing the auxiliary code C2 displayed on each shelf 51 using the terminal 10.

[0269] For example, in step S206 of the above-mentioned item search process, the search unit 343 of the server 30 may refer to the item management table 200 stored in the storage device 32 in response to a search request, and generate item information including a list of items whose positions are registered at the auxiliary coordinates S4 of the auxiliary code C2. Only the item information of items associated with the auxiliary code C2 is displayed on the terminal 100. This can reduce the amount of information processed in the item management system 1.

[0270] Furthermore, the auxiliary code C2 can also be used to correct the position of the terminal 100 within the warehouse 50. The position of the terminal 100 in the reference coordinate system S1 when the image of the auxiliary code C2 is acquired is calculated by the calculation unit 1232. The position of the terminal 100 calculated by the calculation unit 1232 may be compared with the position of the terminal 100 calculated based on the position of the auxiliary code registered in the code management table 100, to correct the position of the terminal 100 within the warehouse 50.

[0271] In the code management table 100, the auxiliary code C2 identified by ref2 to ref4 is displayed on a fixed shelf 51 in the warehouse 50, and is therefore registered as "fixed" in the type column 103. The position and orientation of the auxiliary code C2 are registered in the position column 104 of the code management table 100. The position is expressed in reference coordinates S1, and the orientation is expressed in Euler angles.

[0272] Meanwhile, a movable cart 52 is placed in the warehouse 50, and an item 60 is also placed on this cart. An auxiliary code C3 is displayed on the cart 52. The position and orientation of the auxiliary code C3 change as the cart 52 moves.

[0273] In the code management table 100 shown in FIG. 4(A), the auxiliary code C3 identified by ref5 changes position and direction as the cart 52 moves, so it is registered as "indefinite" in the type column 103.

[0274] The calculation unit 1232 uses the matrix MW5 to convert the position of the article represented by the auxiliary coordinates S4 with the auxiliary code C3 of ref5 as the origin into the position of the reference coordinates S1. The matrix MW5 is also represented by the above formula 12. This matrix MW5 is updated every time the cart 52 moves.

[0275] Next, an example will be described in which the position of an item is calculated using two auxiliary codes C2 (hereinafter also referred to as auxiliary codes C2-1 and C2-2) fixed to different shelves 51. Assume that the camera 113 of the terminal 100 first acquires an image of the auxiliary code C2-1, and then acquires an image of the item 60 whose position is represented by auxiliary coordinates S3 with the auxiliary code C2-1 as the origin.

[0276] Let MW3 be the matrix that converts the position of the article 60 represented by the auxiliary coordinates S3 with the auxiliary code C2-1 as the origin into the position of the reference coordinates S1. Let Mw3 be the matrix that converts the position represented by the coordinates SA with the camera position of the camera 113 at the time the camera 113 is started as the origin into the position represented by the coordinates with the auxiliary code C2-1 as the origin.

[0277] Also, let MW4 be the matrix that converts the position of the item 60 represented in the auxiliary coordinate system S4 with the auxiliary code C2-2 as the origin into the position of the reference coordinate S1. Let M2 be the matrix that converts the position represented by the coordinate SA with the position of the camera 113 at the time the camera 113 is started as the origin into the position of the camera coordinate S2 of the camera 113 of the terminal 100 located at an arbitrary position within the warehouse 50. Here, the arbitrary position within the warehouse 50 is the position of the camera 113 when the image of the item 60 was captured.

[0278] A matrix Mc4 that converts the position of the article 60 represented in the auxiliary coordinate system S4 with the auxiliary code C2-2 as the origin into the position of the camera coordinates S2 of the camera 113 when the image of the article 60 was captured is expressed by the following formula 13.

[0279]

number

[0280] Next, an example of calculating the position of an item using the auxiliary code C2 fixed to the shelf 51 and the auxiliary code C3 displayed on the cart 52 will be described below. Assume that the camera 113 of the terminal 100 first acquires an image of the auxiliary code C2 displayed near the cart 52, and then acquires an image of the item 60 whose position is represented by auxiliary coordinates S3 with the auxiliary code C3 as the origin.

[0281] Let MW5 be the matrix that converts the position of the article 60 represented by the auxiliary coordinates S4 with the auxiliary code C2 as the origin into the position of the reference coordinates S1. Let Mw5 be the matrix that converts the position represented by the coordinates SA with the camera position of the camera 113 at the time the camera 113 is started as the origin into the position of the auxiliary coordinates S4 with the auxiliary code C2 as the origin.

[0282] Also, let Mw6 be the matrix that transforms the position represented by the coordinate SA, which has the camera position of the camera 113 at the time the camera 113 is started as the origin, into the position of the auxiliary coordinate S4, which has the auxiliary code C3 as the origin.

[0283] A matrix MW6 that converts the position of the auxiliary coordinate S4, which has the auxiliary code C3 as its origin, into the position of the reference coordinate S1 is expressed by the following formula 14.

[0284]

number

[0285] The user can easily manage the position of the item 60 placed on the cart 52 by using the terminal 100 to acquire an image of the auxiliary code C2 displayed near the cart 52, and then similarly using the terminal 100 to acquire an image of the auxiliary code C3 displayed on the cart 52.

[0286] In the present invention, the item search device and the item management system of the above-mentioned embodiments can be modified as appropriate without departing from the spirit of the present invention. Furthermore, the technical scope of the present invention is not limited to those embodiments, but covers the inventions described in the claims and their equivalents. Furthermore, the configuration of one embodiment can be applied to other embodiments as appropriate.

[0287] For example, in the above-mentioned search process, the item is searched for using the registered position of the item represented by the reference coordinates and the current position. In this search process, the registered position of the item represented by the reference coordinates and the current position may be expressed as coordinates with the position of the camera when the camera was started as the origin, and the item may be searched for.

[0288] In the above embodiment, the server stores the code management table and the item management table, but the terminal may store the code management table and the item management table. In the above embodiment, the terminal that registers the location of an item and the terminal that searches for the item are different, but one terminal having the functions of the two terminals may be used to register the location of an item and to search for the item.

[0289] In the above-described embodiment, the LiDAR sensor is used as the distance measurement unit, but other sensors may be used as the distance measurement unit. For example, a stereo camera, an RGB-D sensor, or a distance sensor may be used as the distance measurement unit.

[0290] Furthermore, in the above-mentioned item position registration device, the calculation unit may calculate a first position and a first orientation of the position calculation device expressed in a reference coordinate system having an origin at the information code based on an image of the information code acquired by the image acquisition unit and the distance between the image acquisition unit and the information code, calculate a second position representing a current position of the position calculation device expressed in the reference coordinate system and a second orientation representing the current orientation based on the calculated first position and first orientation, position information and angle information, and calculate the position of the item expressed in the reference coordinate system based on an image representing the vicinity of the item acquired by the image acquisition unit, the distance between the image acquisition unit and the vicinity of the item, and the second position and second orientation.

[0291] In addition, in the item search device, when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit generates an extended image to be displayed on the display unit by arranging item information about the item at a position determined based on the current position of the item in the image acquired by the image acquisition unit, and further includes a sensor capable of detecting a distance between the object and the item search device, and when a first distance acquired by the sensor between the item search device and an object located in the direction of the registered position of the item relative to the item search device does not match a second distance between the registered position of the item and the position of the item search device. In this case, the second determination unit determines that the registered position of the item does not match the current position of the item, and the generation unit may, if the first distance is shorter than the second distance, generate an extended image to be displayed on the display unit by arranging information indicating that an object other than the item is located between the item and the item search device as information indicating that the registered position of the item does not match the current position of the item, and if the first distance is longer than the second distance, generate an extended image to be displayed on the display unit by arranging information indicating that the item is not present as information indicating that the registered position of the item does not match the current position of the item.

[0292] In addition, in the item search device, when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit generates an enlarged image to be displayed on the display unit by arranging information indicating that the registered position of the item does not match the current position of the item, and further includes a first calculation unit that calculates the current position of the item represented in reference coordinates based on an image of the item code represented in an image representing the vicinity of the item acquired by the image acquisition unit, the distance between the item search device and the item code, and the current position of the item search device represented in reference coordinates, and when the second determination unit determines that the registered position of the item does not match the current position of the item, the enlarged image generation unit generates an enlarged image by arranging item information regarding the item in a position determined based on the current position of the item in the image acquired by the image acquisition unit.

[0293] In addition, in the item search device, when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit generates an extended image to be displayed on the display unit by arranging information indicating that the registered position of the item does not match the current position of the item; further, the item management table registers registered features of the item depicted in the image together with the registered position of the item, and the device has a second calculation unit that calculates a position change amount indicating information regarding the difference between the registered position of the item and the current position of the item based on the current features of the item depicted in the current image of the item acquired by the image acquisition unit and the registered features of the item; when the position change amount calculated by the second calculation unit is greater than a reference value, the second determination unit determines that the registered position of the item does not match the current position of the item, and the generation unit generates an extended image by arranging item information regarding the item in a position determined based on the current position of the item in the image acquired by the image acquisition unit.

[0294] In addition, in the item search device, when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit generates an extended image to be displayed on the display unit by arranging information indicating that the registered position of the item does not match the current position of the item, and further includes a third calculation unit that calculates the current position of the item based on a point selected on the image representing the item acquired by the image acquisition unit, and calculates the current orientation of the item based on a plane including a point on the item corresponding to the selected point, and when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit may generate an extended image by arranging item information regarding the item in a position determined based on the current position and current orientation of the item in the image acquired by the image acquisition unit. [Explanation of symbols]

[0295] 1. Inventory management system 10 Terminal 11 Communications Department 12 User Interface 13. Camera 14 First Sensor 15 Second Sensor 16 3rd Sensor 17 Materials Management Department 18 Signal Line 21 Communication Interface 22 Memory 23 Processors 231 Control Unit 232 Calculation section 30 Servers 31 Communication Interface 32 Storage Device 33 Memory 34 processors 341 Control Unit 342 Registration Department 343 Search Department 35 Signal Line 40 Communication Network 41 Base station 50 Warehouse 51 Shelf 52 Trolley 60 Goods C1 Reference Code C2 Auxiliary Code C3 Auxiliary Code D1 Item Code D2 Item Code 100 devices 111 Communications Department 112 User Interface 113 Camera 114 First Sensor 115 Second Sensor 116 3rd Sensor 117 Materials Management Department 118 Signal Line 121 Communication Interface 122 Memory 123 processors 1231 Control Unit 1232 Calculation section 1233 Judgment section 1234 Generation part

Claims

1. An object search device that searches for a position of an object represented in a reference coordinate system having an information code as an origin, A display unit; An image acquisition unit; a first determination unit that determines whether or not a registered position of the item, which is represented by the reference coordinates registered in an item management table, is included in a field of view of the image acquisition unit; a second determination unit that, when it is determined by the first determination unit that the registered position of the article is included in the field of view of the image acquisition unit, determines whether or not the registered position of the article coincides with a current position of the article represented by the reference coordinates; a generation unit that, when it is determined by the second determination unit that the registered position of the item and the current position of the item match, places item information about the item in a position determined based on the registered position of the item in the image acquired by the image acquisition unit, and generates an extended image to be displayed on the display unit; An object searching device comprising:

2. The item search device of claim 1, wherein when the second determination unit determines that the registered location of the item does not match the current location of the item, the generation unit places item information about the item at a position determined based on the current location of the item in the image acquired by the image acquisition unit, and generates an augmented image to be displayed on the display unit.

3. The item search device of claim 1, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit generates an augmented image to be displayed on the display unit by arranging information indicating that the registered position of the item does not match the current position of the item.

4. A sensor capable of detecting a distance between an object and the item searching device, If a first distance between the item search device and an object in the direction of the registered position of the item relative to the item search device, which is acquired by the sensor, does not match a second distance between the registered position of the item and the position of the item search device, The second determination unit determines that the registered position of the item does not match the current position of the item, and The item search device of claim 2, wherein when the first distance is shorter than the second distance, the generation unit generates an augmented image to be displayed on the display unit by arranging information indicating that an object other than the item is located between the item and the item search device as information indicating that the registered position of the item does not match the current position of the item, and when the first distance is longer than the second distance, the generation unit generates an augmented image to be displayed on the display unit by arranging information indicating that the item is not present as information indicating that the registered position of the item does not match the current position of the item.

5. the item is provided with an item code; the position of the item code expressed by the reference coordinates is registered in the item management table as a registration position of the item, a first calculation unit that calculates a current position of the item represented by the reference coordinates based on an image of the item code shown in an image showing the vicinity of the item acquired by the image acquisition unit, a distance between the item search device and the item code, and a current position of the item search device represented by the reference coordinates; The item search device of claim 2, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit generates an augmented image by placing item information about the item at a position determined based on the current position of the item in the image acquired by the image acquisition unit.

6. In the item management table, a registered feature amount of the item shown in an image is registered together with a registered position of the item, a second calculation unit that calculates a position change amount representing information relating to a difference between a registered position of the article and a current position of the article based on a current feature amount of the article shown in the current image of the article acquired by the image acquisition unit and a registered feature amount of the article, When the position change amount calculated by the second calculation unit is greater than a reference value, The second determination unit determines that the registered position of the item does not match the current position of the item, and The item search device according to claim 2 , wherein the generation unit generates an augmented image by arranging item information about the item at a position determined based on a current position of the item in the image acquired by the image acquisition unit.

7. a third calculation unit that calculates a current position of the article based on a point selected on an image representing the article acquired by the image acquisition unit, and calculates a current orientation of the article based on a plane including a point on the article corresponding to the selected point; The item search device of claim 2, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit generates an augmented image by placing item information about the item at a position determined based on the current position and current orientation of the item in the image acquired by the image acquisition unit.

8. An item search device as described in any one of claims 5 to 7, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the registered position of the item in the item management table is corrected based on the current position of the item.

9. An item search device as described in any one of claims 5 to 7, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the current orientation of the item search device is corrected based on the current position of the item.

10. A sensor capable of detecting a distance between an object and the item searching device, If a first distance between the item search device and an object in the direction of the registered position of the item relative to the item search device, which is acquired by the sensor, does not match a second distance between the registered position of the item and the position of the item search device, The second determination unit determines that the registered position of the item does not match the current position of the item, and The item search device of claim 3, wherein when the first distance is shorter than the second distance, the generation unit arranges information indicating that an object other than the item is located between the item and the item search device as information indicating that the registered position of the item does not match the current position of the item, and generates an augmented image to be displayed on the display unit, and when the first distance is longer than the second distance, the generation unit arranges information indicating that the item is not present as information indicating that the registered position of the item does not match the current position of the item, and generates an augmented image to be displayed on the display unit.

11. the item is provided with an item code; the position of the item code expressed by the reference coordinates is registered in the item management table as a registration position of the item, a first calculation unit that calculates a current position of the item represented by the reference coordinates based on an image of the item code shown in an image showing the vicinity of the item acquired by the image acquisition unit, a distance between the item search device and the item code, and a current position of the item search device represented by the reference coordinates; The item search device of claim 3, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the enlarged image generation unit generates an enlarged image by placing item information about the item at a position determined based on the current position of the item in the image acquired by the image acquisition unit.

12. In the item management table, a registered feature amount of the item shown in an image is registered together with a registered position of the item, a second calculation unit that calculates a position change amount representing information relating to a difference between a registered position of the article and a current position of the article based on a current feature amount of the article shown in the current image of the article acquired by the image acquisition unit and a registered feature amount of the article, When the position change amount calculated by the second calculation unit is greater than a reference value, The second determination unit determines that the registered position of the item does not match the current position of the item, and The item search device according to claim 3 , wherein the generation unit generates an augmented image by arranging item information about the item at a position determined based on a current position of the item in the image acquired by the image acquisition unit.

13. a third calculation unit that calculates a current position of the article based on a point selected on an image representing the article acquired by the image acquisition unit, and calculates a current orientation of the article based on a plane including a point on the article corresponding to the selected point; The item search device of claim 3, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the generation unit generates an augmented image by placing item information about the item at a position determined based on the current position and current orientation of the item in the image acquired by the image acquisition unit.

14. An item search device according to any one of claims 11 to 13, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the registered position of the item in the item management table is corrected based on the current position of the item.

15. An item search device according to any one of claims 11 to 13, wherein when the second determination unit determines that the registered position of the item does not match the current position of the item, the current orientation of the item search device is corrected based on the current position of the item.

16. An object search device that searches for a position of an object represented in a reference coordinate system having an information code as an origin, A display unit; An image acquisition unit; a first determination unit that determines whether or not a registered position of the item, which is represented by the reference coordinates registered in an item management table, is included in a field of view of the image acquisition unit; a second determination unit that, when it is determined by the first determination unit that the registered position of the article is included in the field of view of the image acquisition unit, determines whether or not the registered position of the article coincides with a current position of the article represented by the reference coordinates; a generation unit that, when it is determined by the second determination unit that the registered position of the item and the current position of the item match, places item information about the item in a position determined based on the registered position of the item in the image acquired by the image acquisition unit, and generates an extended image to be displayed on the display unit; An item searching device including: A storage unit that stores the item management table; An article management system comprising:

17. The item management system described in claim 16, wherein when the second determination unit determines that the registered location of the item does not match the current location of the item, the generation unit places item information about the item at a position determined based on the current location of the item in the image acquired by the image acquisition unit, and generates an augmented image to be displayed on the display unit.

18. The item management system of claim 16, wherein when the second determination unit determines that the registered location of the item does not match the current location of the item, the generation unit generates an augmented image to be displayed on the display unit by arranging information indicating that the registered location of the item does not match the current location of the item.

Images