Administrative system for managing object, server device and program

Abstract

To provide a workload for identifying an object to be managed which is captured in an image.SOLUTION: A server device of an administrative system that manages a plurality of objects, each provided with a first wireless device, determines a time history at a device position of a reading device on the basis of first read data indicating a first reading time read by a reading device from the first wireless device, second read data indicating a second reading time read by the reading device from a second wireless device arranged at a predetermined position, and a moving amount of the reading device. The device determines an object position of the object provided with the first wireless device on the basis of the device position of the reading device at the first read time of the first wireless device, and determines an imaging range of a captured image on the basis of a device position and an imaging direction at an imaging time of a captured image captured by an imaging device related to the reading device, and determines the object where the object position is included in the imaging range.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to a management technique in a management system that manages managed objects in association with images captured of the objects. [Background technology]

[0002] In recent years, as a result of improvements in the performance of information and communication technology, it has become possible to leave images as records in various situations. Patent Document 1 discloses a configuration in which a large number of photos are taken and classified by date or location in order to record the status of construction work that progresses daily. According to Patent Document 1, in order to reduce management errors in the photos taken, the photographing locations are associated with drawings of the construction site and displayed on the screen. Patent Document 2 discloses a configuration in which a worker wears a 360-degree camera and takes images while patrolling the site, and the work status is automatically determined from the images using image recognition technology. Furthermore, Patent Document 2 also discloses performing positioning in parallel with the shooting and associating the location data with the captured images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-95749 A [Patent Document 2] JP 2019-148946 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, consider the case where an image (video) of an object such as an article is captured for management of the object and recorded in association with the object. The image may be a still image or a moving image. In the following description, an object to be managed is referred to as a managed object. When there are a large number of managed objects or when their positions change, the workload of identifying and photographing the managed objects individually increases. Therefore, for example, if a managed object can be identified by image recognition technology from images captured at various positions in the space in which the managed object exists, the workload of associating the managed object with an image can be reduced.

[0005] However, identification of a managed object using image recognition technology can only be applied when the shape and appearance of the managed object are known and the state of the shape and appearance does not change over time. In other words, when the shape or appearance of a managed object is not known or when the state of the managed object changes over time, it is not possible to identify the managed object using image recognition technology.

[0006] In view of the above, the present invention provides a technique for reducing the workload for identifying an object to be managed that appears in an image. [Means for solving the problem]

[0007] According to one aspect of the present invention, a server device of a management system that manages a plurality of objects, each of which is provided with a first wireless device, includes: an acquisition means for acquiring, from the reading device, first read data indicating a first read time at which the reading device reads the first wireless device, and second read data indicating a second read time at which the reading device reads a second wireless device placed at a predetermined position; a first determination means for repeatedly acquiring an amount of movement of the reading device from the reading device, and determining a time history of a device position of the reading device based on the amount of movement of the reading device at the second read time indicated by the second read data and the predetermined position at which the second wireless device was placed; the second determination means for determining an object position of the object to which the first wireless device is attached based on the device position of the reading device at the first reading time of the device; a third determination means for executing a determination process for determining a shooting range of the captured image based on the device position at the shooting time of the image captured by the shooting device associated with the reading device and the shooting direction of the captured image by the shooting device, and for determining the object whose object position is included in the shooting range; and a display means for performing a process of displaying, on the device that requested the execution of the determination process, candidate objects whose object position is all or part of the object included in the shooting range. Effect of the Invention

[0008] According to the present invention, it is possible to reduce the workload for identifying an object to be managed that appears in an image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a management system configuration, according to some embodiments. [Diagram 2] FIG. 1 illustrates an example of a management system configuration, according to some embodiments. [Diagram 3]1 is a block diagram illustrating an example of a tag reader configuration according to some embodiments. [Figure 4] FIG. 2 is a block diagram illustrating an example configuration of an image server according to some embodiments. [Diagram 5] FIG. 2 is a block diagram illustrating an example of a configuration of a management server according to some embodiments. [Figure 6] 4A-4C illustrate examples of a floor table, a map table, and a location table, according to some embodiments. [Figure 7] 1A-1C illustrate examples of a subject table and a reader-camera table, according to some embodiments. [Figure 8] 4A-4C show examples of a movement amount table and a tag detection table according to some embodiments. [Figure 9] 1 is an illustration of an example of determining a position of interest, according to some embodiments. [Figure 10] FIG. 10 is an explanatory diagram of a process performed by a narrowing-down control unit according to some embodiments. [Figure 11] FIG. 10 is an explanatory diagram of a process performed by a narrowing-down control unit according to some embodiments. [Figure 12] 11A-11C show example screens for generating reports according to some embodiments. [Figure 13] 1 is a flow chart of a report generation process according to some embodiments. [Figure 14] 13 is a flowchart of a refinement process according to some embodiments. [Figure 15] 11A-11C show other example screens for creating reports according to some embodiments. [Figure 16] 11A-11C illustrate yet other example screens for generating reports according to some embodiments. [Figure 17] 1 is a flow chart of a report generation process according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] A configuration common to each embodiment will be described below. As shown in Fig. 1 and Fig. 2, the management system according to each embodiment is composed of an image server 150, a management server 200, a tag reader 100, a user terminal 70, a camera 50, a target tag 21, and a position tag 11. The target tag 21 is a general term for the target tags 21a to 21c in Fig. 1. The position tag 11 is a general term for the position tags 11a and 11b in Fig. 1. The image server 150, the management server 200, the tag reader 100, the user terminal 70, and the camera 50 are configured to be able to communicate with each other via a network 5.

[0012] FIG. 1 shows floors 10a and 10b of a building under construction. In the following description, the floors 10a and 10b will be collectively referred to as floor 10. According to FIG. 1, managed objects 20a, 20b, and 20c, which are objects to be managed in the management system, are arranged on the floor 10a. For example, the managed objects 20a and 20b are equipment used for construction work or its management, and the managed object 20c is construction materials. In the following description, the managed objects 20a, 20b, and 20c will be collectively referred to as managed object 20.

[0013] The user 30a is a worker involved in the construction work on the floor 10a. The user 30a periodically patrols the floor 10a and takes pictures of the space with the camera 50. The image data of the images taken by the camera 50 is transmitted to and stored in the image server 150 shown in FIG. 2. The user 30b is a manager responsible for the progress of the construction work. The user 30b can check the progress of the construction work by viewing the images stored in the image server 150 using the user terminal 70. In this embodiment, the users 30a and 30b are different persons, but the users 30a and 30b may be the same person. In the following, the users 30a and 30b are collectively referred to as the user 30.

[0014] The user 30a creates a report on the managed object 20 based on an image of the managed object 20 in order to share with the user 30b where the managed object 20 is located, whether it was used in the correct procedure, whether construction of a specific section was completed as scheduled, and the like. However, the situation on the floor 10a changes over time, and the location and state (status) of the managed object 20 may change. Therefore, the workload of the user 30a to identify and photograph each managed object 20 located on the floor 10a and to identify each managed object 20 from the image taken by the user 30a is large.

[0015] For this reason, in this embodiment, a target tag 21 is used. As the target tag 21, an RFID (Radio Frequency IDentification) tag, which is a wireless device, can be used. In FIG. 1, the target tags 21a, 21b, and 21c are attached to the management objects 20a, 20b, and 20c. Note that the management object 20 in this embodiment is not limited to a movable object. For example, a fixed object such as a ceiling, a floor, or a wall can be the management object 20. Furthermore, a certain section in a real space can be the management object 20 (space object). When a certain section in a real space is the management object 20, the target tag 21 is placed at an arbitrary position in the section which is the management object 20. Also, for example, when a plurality of devices or a plurality of materials are grouped and managed, the group of devices or materials can be one management object 20. In other words, the object to be managed in association with one target tag 21 is one management object 20, and the form of the management object 20 can be arbitrarily determined by the user 30 of the management system.

[0016] The management system further includes a position tag 11, which is an RFID tag. According to FIG. 1, the position tag 11a is placed on the floor 10a, and the position tag 11b is placed on the floor 10b. In the example of FIG. 1, one position tag 11 is placed on each floor, but multiple position tags 11 can be placed on one floor. The placement position of each position tag 11 in real space is a predetermined position and is known in the management system. As described later, the placement position of each position tag 11 is used as a reference position for determining the real space position of the tag reader 100, the real space position of the target tag 21, and the shooting position by the camera 50. As shown in FIG. 1, the user 30a carries the tag reader 100 when shooting a space with the camera 50. The tag reader 100 is a reading device that wirelessly communicates with the position tag 11 and the target tag 21 to read information stored in the position tag 11 and the target tag 21.

[0017] The target tag 21 (first wireless device) and the position tag 11 (second wireless device) are a type of wireless device, and in this embodiment, they are passive RFID tags (passive tags). The passive tag has a small IC (Integrated Circuit) chip with built-in memory and an antenna, and stores identification information (ID) for identifying the passive tag and other information in the memory. In the following description, the identification information stored in the memory of the position tag 11 and the target tag 21 is referred to as a tag ID. The IC chip of the passive tag operates by utilizing the energy of electromagnetic waves radiated from the tag reader 100, and transmits (returns) an information signal based on the information stored in the memory from the antenna.

[0018] The camera 50 is a photographing device that photographs and records images. The camera 50 has a display. An image formed on an image sensor through an optical system such as a lens can be displayed on the display. The camera 50 also has a communication function. The user 30a can transmit image data of an image displayed on the display to the image server 150 in real time by operating the camera 50. The user 30a can record image data of an image displayed on the display in its internal memory by operating the camera 50. The image to be recorded may be a still image (photo) or a moving image (video). The camera 50 can transmit the image data recorded in its internal memory to the image server 150 afterwards. In this embodiment, the camera 50 has a function of detecting the direction in which the image is being photographed (hereinafter, the photographing direction) by, for example, a geomagnetic sensor or the like. Furthermore, in this embodiment, the camera 50 prestores identification information (referred to as a camera ID) that uniquely identifies each camera 50. An image photographed by the camera 50 can be identified by a combination of the camera ID and the photographing time. The image data recorded by the camera 50 includes image management information indicating the camera ID, the shooting time, and the shooting direction in addition to data indicating the image itself. The camera 50 may be a general-purpose terminal such as a smartphone or a mobile phone. Alternatively, the camera 50 may be a dedicated terminal specialized for the purpose of image management. Furthermore, the camera 50 may be a head-mounted display for MR (Mixed Reality).

[0019] 2 is a user device used by a user 30 of the management system and has a display. The user terminal 70 also has a communication function. The user terminal 70 may be, for example, a general-purpose terminal such as a PC (Personal Computer), a smartphone, or a mobile phone, or may be a dedicated terminal specialized for the purpose of image management.

[0020] 1, the camera 50 and the tag reader 100 are physically separate devices, but they may be an integrated device having the functions of both the camera 50 and the tag reader 100. Also, instead of the user 30a wearing and carrying the camera 50 and the tag reader 100, the camera 50 and the tag reader 100 (or an integrated device) may be mounted on a mobile body (e.g., a drone, a vehicle, or a robot) and moved within the real space.

[0021] The image server 150 is an information processing device having a database for storing and accumulating images captured by the camera 50. The management server 200 is an information processing device having a database for storing the results of reading the position tags 11 and the target tags 21 by the tag reader 100. The image server 150 and the management server 200 can be implemented as server devices such as an application server, a database server, or a cloud server using, for example, a high-performance general-purpose computer. An example of a specific configuration of the image server 150 and the management server 200 will be described later.

[0022] The management function of the management server 200, which will be described in detail later, can be provided by a single device, or by multiple physically separate devices working together. The same applies to the image storage function of the image server 150. Furthermore, the image server 150 and the management server 200 may be realized as a single integrated server device. Also, part of the database, which will be described later, may be held in a device separate from the image server 150 and the management server 200 (for example, the camera 50, the tag reader 100, or another server device).

[0023] FIG. 3 is a block diagram showing an example of the configuration of the tag reader 100, which is a reading device according to this embodiment. The control unit 101 has a memory that stores a computer program, and one or more processors (e.g., a CPU (Central Processing Unit)) that execute the computer program. The control unit 101 controls the overall function of the tag reader 100. For example, the control unit 101 causes the reading unit 106 to execute reading of an RFID tag within its reading range, and causes the read information and the read time to be stored as read data in the storage unit 102. The read data includes first read data when the target tag 21 is read, and second read data when the position tag 11 is read. In addition, the control unit 101 causes the measurement unit 104 to measure the amount of movement of the tag reader 100 in parallel with reading the RFID tag, and causes the measurement unit 102 to store the amount of movement information indicating the measurement result and the measurement time as measurement data. Then, the control unit 101 transmits the read data and the measurement data stored in the storage unit 102 to the management server 200 via the communication unit 103 together with the identification information of the own device (referred to as a reader ID).

[0024] The storage unit 102 may include any type of storage medium, such as a semiconductor memory such as a Read Only Memory (ROM) or a Random Access Memory (RAM), an optical disk, or a magnetic disk. In this embodiment, the storage unit 102 stores the above-mentioned read data, measurement data, and the reader ID of the tag reader 100.

[0025] The communication unit 103 is a communication interface for the tag reader 100 to communicate with the management server 200, etc. For example, the communication unit 103 may be a WLAN interface for communicating with a WLAN (Wireless Local Area Network) access point, or a cellular communication interface for communicating with a cellular base station. The communication unit 103 may also be a connection interface for connecting to a relay device (for example, a Bluetooth (registered trademark) interface or a USB (Universal Serial Bus) interface).

[0026] The measurement unit 104 measures the relative movement amount of the tag reader 100 and outputs the measured movement amount to the control unit 101. The measurement unit 104 includes a three-axis acceleration sensor 104a, a gyro sensor 104b, and a geomagnetic sensor 104c. The three-axis acceleration sensor 104a measures the acceleration applied to the tag reader 100 in a device coordinate system specific to the tag reader 100 and outputs first sensor data. The gyro sensor 104b measures the angular velocity of the tag reader 100, i.e., the change in the attitude of the tag reader 100, and outputs second sensor data. The geomagnetic sensor 104c measures the orientation of the tag reader 100 in the real space and outputs third sensor data. The measurement unit 104 can measure the relative movement amount of the tag reader 100 by accumulating the acceleration while converting the direction of the acceleration of the tag reader 100 into a direction in the coordinate system of the real space based on the sensor data from these sensors. The amount of movement here may be measured according to any known self-position estimation method (also called PDR (Pedestrian Dead Reckoning) method). The amount of relative movement output from measurement unit 104 to control unit 101 may be a two-dimensional vector in real space. Alternatively, the amount of relative movement may be a three-dimensional vector in real space including a component in the height direction. The starting point of the measurement of the amount of relative movement (the origin of the device coordinate system) may be, for example, the position of tag reader 100 at the time tag reader 100 is started. Therefore, the amount of relative movement may also be the position coordinate of the device coordinate system used by tag reader 100.

[0027] 3, tag reader 100 includes measuring unit 104, but measuring unit 104 may be included in an external device that is capable of communicating with tag reader 100 and that is carried by the user together with tag reader 100. In this case, tag reader 100 receives movement amount information indicating the relative movement amount from the external device.

[0028] Power supply 105 includes a battery and a DC-DC converter, and supplies power for operating each circuit of tag reader 100. The battery may be a primary battery or a secondary battery. Although not shown, tag reader 100 may have a connection terminal for connecting tag reader 100 to an external power source for charging power supply 105.

[0029] The reading unit 106 reads the tag IDs of the position tag 11 and the target tag 21. As shown in FIG. 3, the RF controller 110 of the reading unit 106 outputs a transmission signal (e.g., a signal modulated in the UHF band) from the TX terminal to the power amplifier 111 under the control of the control unit 101. The power amplifier 111 amplifies the transmission signal input from the RF controller 110 and outputs it to the filter 112. The filter 112 is, for example, a low-pass filter, and removes unnecessary low-frequency components of the transmission signal amplified by the power amplifier 111. The coupler 113 distributes the transmission signal that has passed through the filter 112 to the coupler 114 and the power detection unit 116. The coupler 114 outputs the transmission signal input from the coupler 113 to the antenna 115. The coupler 114 also outputs the reception signal input from the antenna 115 to the RF controller 110.

[0030] The antenna 115 transmits the transmission signal input from the coupler 114 as an electromagnetic wave into the air. The antenna 115 also receives a signal returned from an RFID tag present within the reading range of the tag reader 100 in response to the transmission signal, and outputs the received signal to the coupler 114. The power detection unit 116 detects the power level of the signal input from the coupler 113, and outputs a signal RF_DETECT indicating the detected power level to the control unit 101. The canceller 117 receives a signal CARRIER_CANCEL indicating the power level of the carrier wave from the control unit 101. Then, the canceller 117 cancels the carrier wave component of the reception signal from the antenna 115 at the coupler 114 based on CARRIER_CANCEL, thereby extracting a desired signal component of the reception signal to be output to the RX terminal of the RF controller 110. The RF controller 110 demodulates the signal input from the RX terminal, acquires the tag ID and other information returned from the RFID tag, and outputs the acquired information to the control unit 101. The RF controller 110 also has a function of measuring the strength of the received signal from the position tag 11 and the target tag 21 (hereinafter, referred to as radio wave strength).

[0031] In this embodiment, the reading unit 106 may attempt to read the RFID tag periodically (e.g., once per second) without requiring an explicit instruction from the user. Data transmission from the communication unit 103 to the management server 200 may also be performed periodically (e.g., once every few seconds) without requiring an explicit instruction from the user, or every time an RFID tag is read or a relative movement amount is measured. In order to reduce the communication load by omitting transmission of redundant data, the control unit 101 may exclude from the data to be transmitted a record that is the same as a record that has been transmitted within a recent predetermined period. Note that, in other embodiments, one or both of the attempt to read the RFID tag by the reading unit 106 and the transmission of data to the management server 200 may be performed in response to a user input via some user interface provided in the tag reader 100. When the communication unit 103 communicates with the management server 200 indirectly via a relay device, data can be transmitted to the management server 200 only while the connection between the communication unit 103 and the relay device is valid.

[0032] 4 is a block diagram showing an example of the configuration of the image server 150. The communication unit 160 is a communication interface for the image server 150 to communicate with other devices. The communication unit 160 is a wired communication interface or a wireless communication interface. The image database (DB) 180 is a database that accumulates image data received from the camera 50. The report database (DB) 195 is a database that stores data of reports (report data) created by the user 30.

[0033] Management unit 170 manages image data accumulated in image DB 180 and report data stored in report DB 195. In addition to an image file indicating the image itself, image data includes image management information such as the camera ID of camera 50 that captured the image, the capture time, and the capture direction. In the case of a video, the capture time may include the capture start time and capture end time, or may include the capture start time and duration. When management unit 170 receives an image transmission request specifying some search condition from another device, it searches image DB 180 to extract image management information for one or more images that match the specified search condition, and returns the extracted image management information to the requesting device.

[0034] The playback control unit 190 is a software module that plays back images stored in the image DB 180 (or plays back still images extracted from images). The playback control unit 190 receives a playback request from another device, such as the user terminal 70, via the communication unit 160. The playback request may include information that identifies the image to be played back (e.g., a camera ID and a shooting start time) as well as information that specifies a scene to be played back (e.g., the elapsed time from the shooting start time). In response to receiving such a playback request, the playback control unit 190 extracts the scene to be played back from the image DB 180. Then, the playback control unit 190 sequentially transmits image packets of the extracted scene to the device that made the playback request via the communication unit 160. Note that, if a device that will perform playback is specified in the playback request, the playback control unit 190 may sequentially transmit the image packets to the specified device, not to the device that made the playback request.

[0035] In this specification, a scene means a part of a video. A scene may be a partial video consisting of one or more frames, or a still image. When a playback request requests playback of a still image, the playback control unit 190 may transmit image data of the still image to a device that will perform playback via the communication unit 160, instead of multiple packets including data of the video.

[0036] FIG. 5 is a block diagram showing an example of the configuration of the management server 200. The communication unit 210 is a communication interface for the management server 200 to communicate with other devices. The communication unit 210 is a wired communication interface or a wireless communication interface. The read result DB 220 is made up of a group of tables for managing data indicating the read results of the position tag 11 and the target tag 21 by the tag reader 100. In this embodiment, the read result DB 220 includes a floor table 230, a map table 235, a position tag table 240, a target tag table 250, a reader-camera table 260, a movement amount table 270, and a tag detection table 280. The DB processing unit 290 is a collection of multiple software modules that provide the function of managing the read result DB 220 and performing the processing described below. Each software module can be operated by one or more processors (not shown) of the management server 200 executing a computer program stored in a memory (not shown). The memory here can include a non-transitory computer-readable storage medium. In this embodiment, the DB processing unit 290 includes a registration unit 291, a data management unit 292, a user interface (UI) control unit 293, and a narrowing-down control unit 294.

[0037] FIG. 6(A) shows an example of a floor table 230. The floor table 230 has two data items: a floor ID 231 and a name 232. The floor ID 231 is identification information that uniquely identifies each floor 10. The name 232 indicates the name given to each floor 10. In the example of FIG. 6(A), the name of the floor with the floor ID F01 is Floor A, and the name of the floor with the floor ID F02 is Floor B. In the following description, Floor A corresponds to Floor 10a in FIG. 1, and Floor B corresponds to Floor 10b in FIG. 1.

[0038] FIG. 6B shows an example of the map table 235. The map table 235 has three data items: a floor ID 236, a map image 237, and a scale 238. The floor ID 236 indicates the floor 10 to be associated with the map image in the map table 235 using the value of the floor ID 231 in the floor table 230. The map image 237 is a data item in which map information itself showing a map of the floor 10 identified by the floor ID 236, or link information to the map information is stored. The map information here may typically be image information. The scale 238 indicates a ratio for converting a distance on the map of the map image 237 into a distance in real space (for example, how many meters one pixel on the image corresponds to in real space). The map information shown in the map image 237 may be acquired from an external data source or uploaded by a user and updated at a necessary timing.

[0039] FIG. 6(C) shows an example of the position tag table 240. Each record of the position tag table 240 corresponds to each position tag 11 used in the management system. The position tag table 240 has four data items, namely, a tag ID 241, a floor ID 242, a tag position 243, and a name 244. The tag ID 241 indicates the tag ID of the position tag 11 corresponding to the record. The floor ID 242 indicates the floor 10 on which the position tag 11 corresponding to the record is installed, using the value of the floor ID 231 of the floor table 230. In the example of FIG. 6(C), it is shown that the position tag 11a with the tag ID TG1A is placed on the floor 10a identified by F01, and the position tag 11b with the tag ID TG1B is placed on the floor 10b identified by F02. The tag position 243 indicates the position coordinates in the real space on which the position tag 11 corresponding to the record is installed. The position coordinates may be expressed in the coordinate system of the map information stored in the map image 237, in which case mutual conversion is performed between the coordinate values ​​on the map and the corresponding coordinate values ​​in the real space using the scale 238. The name 244 indicates the name given to the position where the position tag 11 corresponding to the record is installed.

[0040] Fig. 7(A) shows an example of a target tag table 250. Each record in the target tag table 250 corresponds to each target tag 21 used in the management system. The target tag table 250 has three data items: a tag ID 251, a target ID 252, and a name 253. The tag ID 251 indicates the tag ID of the target tag 21 corresponding to the record. The target ID 252 is identification information that uniquely identifies the managed object 20 to which the target tag 21 corresponding to the record is attached, and the name 253 indicates the name given to the managed object 20. The example in Fig. 7(A) shows that a target tag 21 with a tag ID of TG2A is attached to a managed object 20 named Device A.

[0041] In the following description, the target tags 21 with the tag IDs "TG2A", "TG2B", and "TG2C" are assumed to be the target tags 21a, 21b, and 21c in Fig. 1. Therefore, the names of the managed objects 20a, 20b, and 20c in Fig. 1 are device A, device B, and material C, as shown in the target tag table 250. In the following description, the managed objects 20a, 20b, and 20c are also referred to as device A, device B, and material C.

[0042] Fig. 7(B) shows an example of reader-camera table 260. Reader-camera table 260 is a table that holds the association between the reader ID of tag reader 100 and the camera ID of camera 50 that is carried along with tag reader 100 and takes pictures. Reader-camera table 260 has two data items: reader ID 261 and camera ID 262. According to the first record in Fig. 7(B), tag reader 100 with reader ID RD01 is associated with camera 50 with camera ID CM01.

[0043] 8(A) shows an example of the movement amount table 270. The movement amount table 270 is a table for accumulating records of measurement data received from the tag reader 100 (hereinafter, referred to as measurement records). The movement amount table 270 has four data items, namely, a measurement time 271, a reader ID 272, a movement amount 273, and a measurement position 274. The measurement time 271, the reader ID 272, and the movement amount 273 are information included in the measurement data. On the other hand, the measurement position 274 is information determined by the data management unit 292. The measurement time 271 indicates the time when the measurement was performed for the measurement result indicated by each measurement record. The reader ID 272 indicates the reader ID of the tag reader 100 that performed the measurement for the measurement result indicated by each measurement record. The movement amount 273 indicates the relative movement amount measured by the tag reader 100.

[0044] The measurement position 274 is information indicating the real space position (device position) of the tag reader 100 that performed the measurement for the measurement result indicated by each measurement record. The measurement position 274 can also be a two-dimensional coordinate value. As described above, the starting point (origin of the device coordinate system) of the measurement of the relative movement amount is, for example, the position of the tag reader 100 at the time when the tag reader 100 is started. Therefore, the movement amount 273 of the movement amount table 270 is information indicating the position of the tag reader 100 in the device coordinate system with this starting point as the origin. Based on the reading result of the position tag 11 by the tag reader 100, the data management unit 292 converts the position of the tag reader 100's device coordinate system indicated by the movement amount 273 into a real space position and stores it in the measurement position 274 of each measurement record. Note that the measurement position 274 of the measurement record corresponding to the measurement data received before the detection of the position tag 11 is blank and can be updated retroactively after the position tag 11 is detected, for example. In this way, movement amount table 270 is information indicating the time history of the real space position of tag reader 100 (the real space position at each measurement time). Note that the real space position of tag reader 100 at a time other than measurement time 271 can be measurement position 274 in the measurement record of measurement time 271 that is closest to the relevant time. Alternatively, the real space position of tag reader 100 at a time other than measurement time 271 may be estimated by interpolation based on the time history of the real space position of tag reader 100 indicated by movement amount table 270.

[0045] The tag reader 100 detects the position tag 11 within its reading range (detection range). Therefore, the real space position where the tag reader 100 detects the position tag 11 is not a single point, but has a spread corresponding to the reading range. However, in this example, when the tag reader 100 detects the position tag 11, the data management unit 292 determines the measurement position 274 assuming that the tag reader 100 is in the same position as the position tag 11. The data management unit 292 may retroactively update the measurement position 274 every time the tag reader 100 detects the position tag 11. The range of the measurement record to be retroactively updated may be limited. The data management unit 292 may also be configured to determine only the measurement position 274 of the measurement record to be added thereafter, assuming that the position of the tag reader 100 at that time is the position of the position tag 11 when the tag reader 100 detects the position tag 11.

[0046] Note that, although there will be an error in measurement position 274 and detection position 284 (FIG. 8(B)) described later that corresponds to the reading range of tag reader 100, the error will not affect the operation of the management system since the reading range is at most a few meters. Also, based on multiple detection results of the same position tag 11 by tag reader 100, it is possible to further reduce the error in measurement position 274 and detection position 284 described later. The details will be described later.

[0047] FIG. 8B shows an example of the tag detection table 280. The tag detection table 280 is a table for accumulating records of read data received from the tag reader 100 (hereinafter, referred to as read records). The tag detection table 280 has seven data items, namely, a tag ID 281, a read time 282, a reader ID 283, a detection position 284, a target position 285, a target distance 286, and a radio wave intensity 287. The tag ID 281, the read time 282, the reader ID 283, and the radio wave intensity 287 are information included in the read data. The tag ID 281 indicates the tag ID of the position tag 11 or the target tag 21 read by the tag reader 100, and the read time 282 indicates the time of reading. The reader ID 283 indicates the reader ID of the tag reader 100 that has acquired the data of the read record. The radio wave intensity 287 indicates the strength (radio wave intensity) of the signal received from the target tag 21 or the position tag 11 when the tag ID of the target tag 21 or the position tag 11 is read.

[0048] The first to fourth records in Fig. 8(B) are all based on the reading results by the tag reader 100 with a reader ID of RD01. The first record indicates that the position tag 11a with the tag ID of TG1A was read at time T02. The second to fourth records indicate that the target tags 21 with the tag IDs of TG2A, TG2B, and TG2C, that is, the target tags 21 attached to equipment A, equipment B, and material C, were read at time T05.

[0049] The detection position 284 indicates the real space position of the tag reader 100 at the time when the target tag 21 or the position tag 11 is read. The data management unit 292 determines the real space position of the tag reader 100 at the read time 282 based on the history of the measurement position 274 of the movement amount table 270 shown in FIG. 8(A) and stores it in the detection position 284. According to FIG. 8(B), the target tags 21 attached to the device A, the device B, and the material C are read by the tag reader 100 at the position (U5, V5) at the same time T05. As described above, the detection position 284 at the read time 282 may be the measurement position 274 at the measurement time 271 closest to the read time 282. Alternatively, the detection position 284 at the read time 282 may be a position estimated by interpolation based on the time history of the real space position of the tag reader 100 shown in the movement amount table 270.

[0050] Target distance 286 indicates the distance from tag reader 100 to target tag 21 when tag reader 100 detects target tag 21. Data management unit 292 determines the distance from tag reader 100 to target tag 21 from the value of radio wave intensity 287, and stores it in target distance 286. Note that a configuration may also be adopted in which tag reader 100 determines the distance to target tag 21 based on the radio wave intensity, and includes the determined distance in the read data.

[0051] Target position 285 indicates the real space position (object position) of target tag 21 when tag reader 100 detects target tag 21. Data management unit 292 determines the real space position of target tag 21 using a method described later, and stores it in target position 285. In this way, tag detection table 280 is information that indicates the real space position of target tag 21.

[0052] Next, a description will be given of the processing performed by the DB processing unit 290 of the management server 200. The contents of the floor table 230, the map table 235, the position tag table 240, and the target tag table 250 can be determined in advance by a user. The registration unit 291 accepts user input of the determined table contents, and registers each of the contents in the corresponding table.

[0053] The association between the reader ID and the camera ID in the reader-camera table 260 may also be determined in advance by the user and registered in advance. Alternatively, the association between the reader ID and the camera ID may be dynamically recognized when a tag reading attempt is made. For example, a third type of RFID tag (hereinafter referred to as a camera tag) different from the position tag 11 and the target tag 21 is attached to the camera 50. The tag reader 100 reads the identification information (camera ID) from this camera tag and transmits the read data together with the reader ID of the own device to the management server 200. When the received read data includes a read result for the camera tag, the registration unit 291 may register in the reader-camera table 260 an association between the reader ID of the tag reader 100 that is the sender of the read data and the camera ID indicated by the read result. In this configuration, the reader-camera table 260 may include an additional data item indicating the validity period of each association (for example, from the detection time of the camera tag to the end time of the read attempt). This makes it easy to change the combination of camera 50 and tag reader 100.

[0054] The data management unit 292 adds a measurement record to the movement amount table 270 based on the measurement data repeatedly received from the tag reader 100 via the communication unit 210. As described above, the measurement data includes movement amount information indicating the relative movement amount of the tag reader 100 measured by the measurement unit 104 of the tag reader 100. The data management unit 292 records the relative movement amount in the movement amount 273 column of the movement amount table 270. In addition, the data management unit 292 determines the relationship between the device coordinate system of the tag reader 100 and the world coordinate system of the real space by detecting the position tag 11, and records the real space position of the tag reader 100 at each measurement time in the measurement position 274. In this way, the data management unit 292 functions as a first determination unit that determines and records the time history of the real space position of the tag reader 100.

[0055] The data management unit 292 also functions as an acquisition unit that acquires read data from the tag reader 100. The read data includes first read data which is the read result of the target tag 21 and second read data which is the read result of the position tag 11. The data management unit 292 adds a read record to the tag detection table 280 based on the read data. As described above, the read data includes the tag IDs of the target tag 21 and the position tag 11 read by the tag reader 100. Based on the read time 282 indicated by the read record, the data management unit 292 determines the real space position of the tag reader 100 that transmitted the second read data that is the basis of the read record at the read time 282, and adds it to the detection position 284 column.

[0056] Furthermore, the data management unit 292 inputs a value to the target distance 286 based on the value of the radio wave intensity 287. Furthermore, as described below, the data management unit 292 determines the real space position of the target tag 21, that is, the real space position of the managed object 20, and records it in the target position 285 field. In this way, the data management unit 292 also functions as a second determination unit that determines the real space position of the managed object 20. An example of a method for determining the target position 285 will be described below with reference to FIG. 9.

[0057] 9, the tag reader 100 detects the same target tag 21 provided in the same managed object 20 at both time T1 and time T2 after time T1. In this case, it can be determined that there is a high probability that the target tag 21 exists in an overlapping area 302 where the reading range 300 of the tag reader 100 at time T1 and the reading range 301 of the tag reader 100 at time T2 overlap. Note that the reading range 300 of the tag reader 100 at time T1 is determined based on the real space position of the tag reader 100 at time T1, and the reading range 301 of the tag reader 100 at time T2 is determined based on the real space position of the tag reader 100 at time T2. The data management unit 292 can determine any position within the overlapping area 302 as the real space position of the target tag 21 and record it in the target position 285. For example, the data management unit 292 can determine the center position, center of gravity position, or the position that is the shortest distance from the tag reader 100 to the overlapping area 302 within the overlapping area 302 as the real space position of the target tag 21.

[0058] Furthermore, the data management unit 292 can determine the real space position of the target tag 21 within the overlap area 302, the position that is away from the tag reader 100 by the object distance 286. Furthermore, the data management unit 292 can determine the direction from the tag reader 100 to the target tag 21 based on the overlap area 302, and can determine the real space position of the target tag 21, the position that is away from the tag reader 100 by the object distance 286 in the determined direction. In this case, the determined real space position of the target tag 21 may not be within the overlap area 302. Also, three or more detection results can be used instead of two detection results. By using many detection results, the overlap area 302 can be narrowed down further, or the range of the direction from the tag reader 100 to the target tag 21 can be narrowed down, and the determination accuracy of the target position 285 can be improved.

[0059] In this embodiment, the real space position of the target tag 21 is determined based on two or more detection results of the same target tag 21. However, if the reading range of the tag reader 100 is not wide, the detection position of the tag reader 100 when the target tag 21 is detected may be set as the real space position of the target tag 21. In this case, the real space position of the target tag 21 can be determined based on a single detection result of the target tag 21.

[0060] The method described in Fig. 9 can also be applied to the detection of the position tag 11. As described above, when the tag reader 100 detects the position tag 11, the data management unit 292 determines the measurement position 274 assuming that the tag reader 100 is in the same position as the position tag 11. However, the data management unit 292 can perform the same process as in Fig. 9 based on a plurality of detection results at different reading times of the same position tag 11 by the tag reader 100 to accurately determine the positional relationship between the tag reader 100 and the position tag 11. Therefore, the data management unit 292 can more accurately determine the relationship between the device coordinate system and the world coordinate system, and therefore can more accurately determine the value to be stored in the measurement position 274 of the movement amount table 270.

[0061] First Embodiment Next, a first embodiment will be described. In this embodiment, when a user 30 photographs a real space with a camera 50 and the photographed image is displayed on the display of the camera 50, all or some of the managed objects 20 included in the photographed image are presented to the user 30 as "candidate objects". The user 30 selects a "creation object" for which a report is to be created from the presented candidate objects and creates a report. In the following description, an image of the real space currently being photographed by the camera 50 is referred to as a "real-time image".

[0062] FIG. 10 shows a state where the user 30a is shooting while standing still at time T05. The user 30a uses a camera 50 with a camera ID of CM01 and shoots in a shooting direction Y. At this time, the shooting range 310 of the real space where the camera 50 is shooting can be determined based on the real space position (shooting position) of the camera 50 at time T05, the shooting direction Y at time T05, and the camera parameters (internal parameters) of the camera 50. The internal parameters are optical parameters such as the angle of view of the camera 50. In this embodiment, the camera 50 transmits information indicating the shooting direction Y to the management server 200 while shooting a real-time image. According to FIG. 10, the device A is outside the shooting range 310, and the device B and the material C are inside the shooting range 310. That is, the real-time image at time T05 shows the device B and the material C, but not the device A. The tag reader 100 reads the target tags 21 of the device A, the device B, and the material C, and transmits the read results to the management server 200. Based on the read results, the management server 200 adds the second to fourth records to the tag detection table 280 in FIG. 8(B).

[0063] The narrowing control unit 294 determines the photographing range 310 based on the camera parameters of the camera 50, the photographing direction Y at time T05, which is the current time, and the photographing position at time T05. The photographing position at time T05 is determined from the measurement position 274 of the movement amount table 270. The camera parameters of the camera 50 are stored in advance in the management server 200. The photographing direction Y is received from the camera 50 as described above. The narrowing control unit 294 determines that the real-time image includes the target tag 21 of the record whose target position 285 is included in the photographing range 310 among the read records at time T05 and within a predetermined period in the past from time T05 in the tag detection table 280. Therefore, the narrowing control unit 294 determines that the managed object 20 to which the determined target tag 21 is attached is shown in the real-time image. In the example of FIG. 10, it is determined that the equipment B and the material C are shown in the real-time image. In this case, the narrowing-down control unit 294 determines that the device B and the material C are the candidate objects. In this way, the narrowing-down control unit 294 functions as a third determination unit that determines the candidate objects that are the managed objects 20 included in the shooting range 310.

[0064] Furthermore, the narrowing control unit 294 can further narrow down the managed objects 20 to be candidate objects from the managed objects 20 included in the real-time image. FIG. 11 is an explanatory diagram of a process of narrowing down the managed objects 20 to be candidate objects when a plurality of managed objects 20 are included in the real-time image. FIG. 11(A) shows a real-time image at time T05. The selected point P in FIG. 11(A) may be a focus point focused by the autofocus of the camera 50, or a point selected by the user 30 on the display of the camera 50. In addition, when the camera 50 is a head-mounted display for MR, the selected point P may be a point at the line of sight of the user 30 wearing the head-mounted display for MR. The camera 50 transmits a judgment distance A to the selected point P to the management server 200. The judgment distance A may be determined by a distance measurement method using, for example, LiDAR (Light Detection and Ranging). Alternatively, the judgment distance A may be determined by a distance measurement method using SLAM (Simultaneous Localization and Mapping) or image recognition.

[0065] Distances LB and LC in FIG. 11B are distances to equipment B and material C in the photographing range 310, which are indicated in the read record at time T05 in the tag detection table 280. The narrowing-down control unit 294 can further narrow down the managed objects 20 included in the real-time image to be candidate objects by comparing the judgment distance A with the distances LB and LC. For example, the narrowing-down control unit 294 can set the managed objects 20 whose difference from the judgment distance A is within a predetermined value as the candidate objects. Alternatively, the narrowing-down control unit 294 can set a predetermined number of managed objects 20 as candidate objects in order of the smallest difference from the judgment distance A. Note that the method of narrowing down the candidate objects is not limited to the above-mentioned method, and for example, a predetermined number of managed objects 20 can be set as candidate objects in order of the largest value of the radio wave intensity 287 (descending order).

[0066] Fig. 12 shows an example of a user interface (UI) for creating a report according to this embodiment. In response to a report creation request from the camera 50, the UI shown in Fig. 12 is displayed on the display of the camera 50 by the UI control unit 293. The title of the report to be created is input into a title input field 501 of the report creation screen 500. The name of the user creating the report is input into a report creator field 502. The report creation date field 503 is input with the date the report was created. In an image display area 504, a real-time image captured by the camera 50 is displayed.

[0067] The floor display field 505 displays the floor on which the image is captured. The UI control unit 293 can determine the floor based on the position of the tag reader 100 associated with the camera 50. The candidate selection field 506 displays a list 510 of candidate objects determined by the narrowing control unit 294. In FIG. 12, the equipment B is displayed as candidate 1, and the material C is displayed as candidate 2. The display order in the list 510 can be determined, for example, in the narrowing process. For example, the candidate objects can be displayed in the order of the smallest difference from the judgment distance A. Alternatively, the candidate objects can be displayed in the order of the largest value of the radio wave intensity 287. The list 510 is configured to be able to accept a user input to display a managed object 20 that is included in the real-time image but is not included in the candidate objects. In FIG. 12, the list of "display other candidates" corresponds to this, and when the user 30 selects this list, a list of managed objects 20 that are included in the real-time image but are not included in the candidate objects is displayed.

[0068] Also, when the user 30 selects the list "Show other candidates", the control unit 294 may display the managed object 20 that is determined not to be included in the real-time image. The displayed managed object 20 may be, for example, a managed object 20 to which a target tag 21 is attached that is indicated in the tag detection table 280 as having been read by the tag reader 100 within a predetermined period of time from the current time. For example, in the example of FIG. 10, the control unit 294 determines that the device A is not included in the real-time image, but the target tag 21 is within the reading range of the tag reader 100. For example, due to an error in the detection position 284 or an error in the shooting direction Y, the control unit 294 may determine that a managed object 20 that is actually included in the real-time image is not included in the real-time image. In such a case, the user 30 may recognize that the managed object 20 is displayed on the display of the camera 50. In such a case, the user 30 can select the list "Show other candidates" to display managed objects 20 that have been determined by the narrowing-down control unit 294 not to be included in the real-time image.

[0069] The user 30 can select a creation object, which is a management object 20 for creating a report, from the list 510. When the user 30 selects a creation object, the creation object is associated with an image including the creation object. As shown in Fig. 12, the UI control unit 293 can display an icon indicating radio wave strength for each candidate object in the list 510 based on the value of the radio wave strength 287 in the tag detection table 280. The user 30 can use the radio wave strength as reference information when selecting a creation object from the list 510.

[0070] When the user 30 operates the shooting button 507, the camera 50 records the image displayed in the image display area 504. In addition, the user 30 can select the status of the created object from a list displayed by operating the status selection field 508. The selected status is recorded in the report. The list displayed by operating the status selection field 508 is set in advance in the UI control unit 293 in consideration of the purpose of management. When the user 30 operates the completion button 509, the camera 50 transmits image data of the image recorded by operating the shooting button 507 to the image server 150. In addition, when the user 30 operates the completion button 509, the camera 50 or the UI control unit 293 transmits report data of a report including each information inputted in the report creation screen 500 to the image server 150. The report includes information identifying the image recorded by operating the shooting button 507. The image server 150 stores the image data received from the camera 50 in the image DB 180, and stores the report data received from the camera 50 or the management server 200 in the report DB 195. The report data may include image data associated with the report indicated by the report data. In this case, the report data including the image data of the image recorded by operating the capture button 507 is stored in the report DB 195.

[0071] When many managed objects 20 exist within the detection range of the tag reader 100, it is not easy to identify the managed objects 20 that appear in the real-time image. For this reason, in this embodiment, the management server 200 determines the managed objects 20 that appear in the real-time image and displays them as candidate objects on the display of the camera 50. This reduces the workload for determining the managed objects 20 that appear in the real-time image. Therefore, the user 30 can easily select the object to be created from the candidate objects, reducing selection errors and facilitating management work.

[0072] The UI described with reference to FIG. 12 can typically be realized by the UI control unit 293 cooperating with the camera 50. For example, the report creation screen 500 can be a web page displayed by a web browser running on the camera 50. It may also be an application screen displayed by an application downloaded to the camera 50 and executed. The processor of the camera 50 that executes a computer program for such a web browser or application may also be considered to be a control unit that controls the UI. The UI may also be realized by cooperating with the user terminal 70.

[0073] Fig. 13 is a flowchart of the report creation process. In S10, UI control unit 293 of management server 200 causes report creation screen 500 as shown in Fig. 12 to be displayed on the display of camera 50. In S11, camera 50 displays real-time images being captured in image display area 504. In S12, UI control unit 293 displays a floor (location of image capture) determined based on the position of tag reader 100 associated with camera 50 in floor display field 505. The position of tag reader 100 is determined from movement amount table 270, and the floor is determined from floor table 230 and map table 235.

[0074] In S13, the UI control unit 293 displays a list 510 of candidate objects determined by the narrowing control unit 294 in the candidate selection field 506. In this way, the UI control unit 293 and the camera 50 function as a display unit that performs processing to display the candidate objects on the display of the camera 50. In S14, the user 300 inputs a selection in the candidate selection field 506. In S15, when the user 30 operates the shooting button 507, the camera 50 records image data of the image displayed on the display. In S16, the UI control unit 293 accepts a selection in the status selection field 508 by the user 30. In S17, when the user 30 operates the completion button 509, the camera 50 transmits image data to the image server 150, and the UI control unit 293 transmits report data to the image server 150. Note that instead of the camera 50 transmitting image data to the image server 150, the UI control unit 293 may acquire image data from the camera 50 and transmit it to the image server 150 together with the report data. Furthermore, the camera 50 may be configured to transmit image data and report data to the image server 150. In this manner, one of the UI control unit 293 and the camera 50 functions as a recording unit that associates a created object with a captured image and records them in the image server 150. Alternatively, the UI control unit 293 and the camera 50 cooperate to function as a recording unit that associates a created object with a captured image and records them in the image server 150.

[0075] 14 is a flowchart of the candidate object determination process executed by the narrowing-down control unit 294. In S20, the narrowing-down control unit 294 determines the shooting range 310 based on the shooting direction Y received from the camera 50, the real space position of the camera 50, and the internal parameters of the camera 50. In S21, the narrowing-down control unit 294 refers to the tag detection table 280 to determine the target tags 21, and therefore the managed objects 20, within the shooting range.

[0076] In S22, the narrowing control unit 294 judges whether to set all the managed objects 20 within the shooting range as candidate objects or to narrow down further. For example, if the number of managed objects 20 within the shooting range is greater than a threshold, narrowing down is performed to reduce the number of managed objects 20 within the shooting range to the threshold or below the threshold, and in other cases, narrowing down is not performed. Also, as described in FIG. 11, it is also possible to narrow down the candidate objects to managed objects 20 whose difference from the judgment distance A is within a predetermined value. Alternatively, it is also possible to narrow down the candidate objects to managed objects 20 to which a predetermined number of target tags 21 are attached in descending order of radio wave intensity.

[0077] If S22 is Yes, the narrowing control unit 294 performs narrowing processing of the managed objects 20 within the shooting range 310 to determine candidate objects. On the other hand, if S22 is No, the narrowing control unit 294 determines all of the managed objects 20 within the shooting range 310 as candidate objects.

[0078] <Modification> In the above, the management server 200 presents the candidate object to the user 30 based on the real-time image captured by the camera 50. However, it is also possible to configure the management server 200 to present the candidate object to the user 30 when the image of the image data captured by the camera 50 and stored in its memory is displayed on the display of the camera 50. In this case, the camera 50 extracts the shooting time of the captured image displayed on the display and the shooting direction Y at the shooting time from the image data of the captured image displayed on the display and transmits them to the management server 200. The management server 200 determines the shooting position by referring to the movement amount table 270 based on the shooting time of the captured image, and thereby determines the shooting range 310. In addition, the management server 200 determines the candidate object by referring to the tag detection table 280 based on the shooting time of the captured image and determining the target tag 21 detected near the shooting time. In this case, by operating the shooting button 507 in FIG. 12, the image data of the captured image displayed on the display is recorded for a report, and the image data is transmitted to the image server 150 by operating the completion button 509.

[0079] In this embodiment, the camera 50 transmits the shooting direction Y to the management server 200 for the narrowing down process. However, when the camera 50 and the tag reader 100 are integrated into one device, the shooting direction Y can be determined by the measurement unit 104 of the tag reader 100. Therefore, in such a case, the tag reader 100 may be configured to include the direction of the optical axis of the camera 50 (i.e., the shooting direction Y) in the measurement record and transmit it to the management server 200. In this case, the data management unit 292 records the optical axis direction in each record of the movement amount table 270, and the narrowing down control unit 294 can determine the shooting range 310 by taking the optical axis direction of the movement amount table 270 as the shooting direction Y. Also, even if the camera 50 and the tag reader 100 are separate devices, the shooting direction Y can be derived based on the direction detected by the tag reader 100 by connecting the relative relationship between the camera 50 and the tag reader 100 in a fixed manner. Also, the camera 50 may repeatedly transmit the optical axis direction to the management server 200 as optical axis data, and the relationship between the time and the optical axis direction may be recorded in a newly provided optical axis table in the management server 200. In this case, the narrowing control unit 294 determines the shooting direction Y based on the optical axis table.

[0080] <Summary> As described above, according to this embodiment, the candidate objects are presented to the user 30 based on the captured image displayed on the photographing device (camera 50). The candidate objects are all or part of the managed objects 20 included in the captured image displayed on the photographing device. By presenting the managed objects 20 included in the captured image as candidate objects to the user 30, the workload of identifying the managed objects 20 appearing in the captured image is reduced. Furthermore, it becomes easier to associate the target of report creation with the image including the target. Since this embodiment does not require the managed object 20 to be identified by image recognition, it functions effectively even in a situation where the shape of the managed object 20 may change over time. Furthermore, since this embodiment does not require positioning by radio waves from an external source such as a GPS satellite or a base station, it is also possible to apply it to management of construction work carried out inside or underground of a building, for example.

[0081] Second Embodiment Next, the second embodiment will be described, focusing on the differences from the first embodiment. In the first embodiment, a candidate object was determined based on an image displayed on the display of the camera 50, and presented to the user 30. Note that the captured image displayed on the display of the camera 50 was an image at that time, or an image of image data recorded by the camera 50 and stored in its memory. In this embodiment, a candidate object is determined based on image data captured by the camera 50 and transmitted to the image server 150, and stored in the image DB 180 of the image server 150, and presented to the user 30.

[0082] 15 shows an example of a target selection screen 550 that the UI control unit 293 displays on the display of the user terminal 70 in response to a report creation request from the user terminal 70. In a time period selection field 551, the user 30 inputs the time period in which the image that the user wishes to view was taken. In a floor selection field 552, the user 30 inputs the location in which the image that the user wishes to view was taken. In this example, the location in which the image was taken is distinguished by floor 10. The UI control unit 293 displays a map of the floor 10 selected in the floor selection field 552 in a map display area 553. The UI control unit 293 can acquire map information representing the map of the floor 10 by referring to the map table 235.

[0083] Furthermore, the UI control unit 293 performs a display process of superimposing and displaying an icon (black circle in the figure) of the managed object 20 corresponding to the target tag 21 detected on the floor 10 input in the floor selection field 552 during the time period input in the time period selection field 551 on the map image in the map display area 553. The display position of the icon is displayed according to the target position 285 of the tag detection table 280, and multiple managed objects 20 whose target positions 285 are within a predetermined distance can be displayed as one icon. In the example of FIG. 15, the equipment A is displayed as one icon, and the equipment B and materials C are displayed as one icon. The user 30 can select (for example, touch or click) one of the icons of the managed objects 20. When the next button 554 is operated, the UI control unit 293 displays a report creation screen 500 related to the managed object 20 corresponding to the icon selected at that time on the display of the user terminal 70.

[0084] FIG. 16 shows an example of a report creation screen 500 of this embodiment. The same elements as those in FIG. 12 are given the same reference numerals, and differences from the first embodiment will be described. The UI control unit 293 performs processing for displaying, in the image display area 504, an image taken on the floor 10 selected in the floor selection field 552 during the time period inputted in the time period selection field 551 of FIG. 15. For example, the UI control unit 293 acquires image data of an image taken on the floor 10 selected in the floor selection field 552 during the time period inputted in the time period selection field 551 of FIG. 15 from the image server 150 and displays the image in the image display area 504. Alternatively, the UI control unit 293 instructs the image server 150 to transmit an image taken on the floor 10 selected in the floor selection field 552 during the time period inputted in the time period selection field 551 of FIG. 15 to the user terminal 70 and display the image. The UI control unit 293 controls so that, among the images in the time period inputted in the time period selection field 551, an image at the time when the managed object 20 selected by the user 30 in the target selection screen 550 was detected is displayed in a cue-up manner. When the managed object 20 selected by the user 30 in the target selection screen 550 is detected multiple times in the time period inputted in the time period selection field 551, the UI control unit 293 can control so that the earliest detection time is displayed in a cue-up manner. The playback time bar 580 indicates the cue-up time. The user 30 can play back the images (videos) taken in the time period inputted in the time period selection field 551 by operating the playback button 577. Furthermore, the user 30 can display, in the image display area 504, an image taken at any time in the time period inputted in the time period selection field 551 by operating the playback time bar 580.

[0085] In the candidate selection field 506, a list 510 of candidate objects determined by the narrowing control unit 294 is displayed. The narrowing control unit 294 determines the shooting position of the camera 50 based on, for example, the shooting time of the captured image currently displayed in the image display area 504. Then, the narrowing control unit 294 determines the candidate object in the same manner as in the first embodiment based on the shooting direction Y at the shooting time included in the image data acquired from the image server 150, the determined shooting position, and the internal parameters of the camera 50. As in the first embodiment, the user selects a created object, which is a managed object 20 for creating a report, from the list 510. As a result, the created object is associated with the captured image including the created object.

[0086] 16, the captured image to be associated with the created object can be the captured image (moving image) itself based on image data obtained from the image server 150. Alternatively, the captured image to be associated with the created object can be the captured image (still image) currently displayed in the image display area 504. Furthermore, the report creation screen 500 can be configured to accept a user input specifying a cropping range of the captured image, and a part of the captured image obtained from the image server 150 can be cropped and the cropped captured image (moving image) can be associated with the created object.

[0087] When the user 30 operates the completion button 509, the UI control unit 293 transmits report data of the report including each piece of information inputted on the report creation screen 500 and image data of the captured image associated with the report to the image server 150. The report includes image data of the captured image associated with the report, or includes information for identifying the image data of the captured image associated with the report. The image server 150 stores the image data in the image DB 180, and stores the report data in the report DB 195. Alternatively, the image server 150 stores the report data including the image data in the report DB 195.

[0088] Fig. 17 is a flowchart of the report creation process according to this embodiment. In S30, the UI control unit 293 of the management server 200 causes the user terminal 70 to display a target selection screen 550 as shown in Fig. 15. In S31, the UI control unit 293 accepts inputs to the time period selection field 551 and floor selection field 552, that is, inputs of the image shooting time period and shooting location. In S32, the UI control unit 293 displays a map of the input shooting location in the map display area 553, and displays the managed object 20 detected during the shooting time period as an icon in the map display area 553. In S33, the UI control unit 293 accepts a selection input of the icon of the managed object 20.

[0089] When the user 30 operates the next button 554, the UI control unit 293 displays the report creation screen 500 on the display of the user terminal 70 in S34 based on the icon of the managed object 20 selected at that time (hereinafter, selected managed object). In S35, the UI control unit 293 displays in an image display area 504 of the report creation screen 500 the image at the read time of the target tag 21 attached to the selected managed object.

[0090] In S36, the UI control unit 293 displays a list 510 of candidate objects determined by the narrowing control unit 294 in the candidate selection field 506. In S37, the UI control unit 293 accepts a selection input in the candidate selection field 506 by the user 30, and in S38, accepts a selection input in the status selection field 508 by the user 30. In S39, when the user 30 operates the completion button 509, the UI control unit 293 transmits the report data and image data to the image server 150. Note that the user terminal 70 may transmit the report data and image data to the image server 150, instead of the UI control unit 293. In this manner, one of the UI control unit 293 and the user terminal 70 functions as a recording unit that associates a created object with a photographed image and records the associated object in the image server 150. Alternatively, the UI control unit 293 and the user terminal 70 cooperate with each other to function as a recording unit that associates a created object with a photographed image and records the associated object in the image server 150.

[0091] <Summary> As described above, according to this embodiment, the candidate objects are presented to the user 30 based on the image data captured by the imaging device (camera 50) and stored in the image server 150. More specifically, when the user displays the captured image specified by the time period or location of the image on the user terminal 70, all or some of the managed objects 20 included in the captured image displayed on the user terminal 70 are presented to the user 30 as candidate objects. This configuration can reduce the workload of the user 30 in identifying the managed object 20 shown in the captured image and determining the captured image to be associated with the report to be created. This embodiment also does not require the managed object 20 to be identified by image recognition, so it functions effectively even in a situation where the shape of the managed object 20 may change over time. Furthermore, this embodiment also does not require positioning by radio waves from an external source such as a GPS satellite or a base station, so it can be applied to management of construction work carried out inside or underground of a building, for example.

[0092] [Other embodiments] Note that the embodiments have been explained using the example of the progress of construction work on a building under construction, but the management systems according to each embodiment are not limited to application to the management of items, etc. in building construction work, and can be applied in situations where multiple objects need to be managed.

[0093] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0094] The disclosure of this embodiment includes the following configuration. (Configuration 1) A server apparatus of a management system that manages a plurality of objects, each of which is provided with a first wireless device, an acquiring means for acquiring, from the reading device, first read data indicating a first read time when the reading device reads the first wireless device, and second read data indicating a second read time when the reading device reads a second wireless device disposed at a predetermined position; a first determination means for repeatedly acquiring a movement amount of the reading device from the reading device, and determining a time history of a device position of the reading device based on the movement amount of the reading device at the second reading time indicated by the second reading data and the predetermined position where the second wireless device is placed; a second determination means for determining an object position of an object in which the first wireless device is provided, based on the device position of the reading device at the first reading time of the first wireless device indicated by the first reading data; a third determination means for executing a determination process of determining a photographing range of a photographed image based on a device position at a photographing time of the photographed image photographed by the photographing device associated with the reading device and a photographing direction of the photographed image by the photographing device, and determining that the object position is included in the photographing range; a display unit that performs processing to display, on a device that has requested execution of the determination processing, candidate objects that are all or part of the object whose object position is included in the shooting range; A server device comprising: (Configuration 2) the device that has requested the execution of the determination process is the photographing device, the captured image is an image currently being captured by the photographing device, the photographing time is the current time, 2. The server apparatus according to configuration 1, wherein the third determination means acquires the shooting direction from the shooting device. (Configuration 3) the device that has requested the execution of the determination process is the photographing device, The captured image is an image of image data captured by the imaging device and stored in the imaging device, 2. The server apparatus according to configuration 1, wherein the third determination means acquires the shooting direction and the shooting time from the shooting device. (Configuration 4) the device that has requested the execution of the determination process is a user device different from the photographing device, the captured image is an image represented by image data captured by the imaging device and stored in the server device or another server device, 2. The server device according to configuration 1, wherein the third determination means acquires the shooting direction and the shooting time from the image data stored in the server device or the other server device. (Configuration 5) The display means includes: displaying, on the user device, one or more first wireless devices read by the reading device during a time period designated by the user device; The server device of configuration 4, when at least one first wireless device among the one or more first wireless devices is selected by the user device, performs a display process to display on the user device an image of the first reading time during the time period when the at least one first wireless device was read by the reading device. (Configuration 6) The server device according to configuration 5, wherein the display means performs a process of transmitting the image data for the time period stored in the server device or the other server device to the user device, and performs the display process based on the image data for the time period. (Configuration 7) The image data stored in the server device or the other server device is associated with information indicating a shooting location, The display means includes: displaying, on the user device, one or more first wireless devices read by the reading device at a time and place specified by the user device; The server device of configuration 4, when at least one first wireless device among the one or more first wireless devices is selected by the user device, performs a display process to display on the user device an image of the first reading time during the time period when the at least one first wireless device was read by the reading device. (Configuration 8) The server device described in configuration 7, wherein the display means performs a process of transmitting the image data for the time period associated with the location, which is stored in the server device or the other server device, to the user device, and performs the display process based on the image data for the time period associated with the location. (Configuration 9) The server device according to any one of configurations 1 to 8, wherein the third determination means acquires information indicating a determination distance from the device that requested execution of the determination process, and determines, as the candidate object, an object whose object position is included in the shooting range and whose difference between the object position and the determination distance is within a predetermined value. (Configuration 10) the first read data includes information indicating radio wave intensity from the first wireless device when the reading device reads the first wireless device; The server device according to any one of configurations 1 to 8, wherein the third determination means determines a predetermined number of the objects as the candidate objects in descending order of the radio wave strength from the first wireless device provided on the object whose object position is included in the shooting range. (Configuration 11) A server device described in any one of configurations 1 to 10, wherein the second determination means determines each of a plurality of device positions at different first read times based on the first read data indicating that the reading device read the same first wireless device at different first read times, and determines the object position of the object in which the same first wireless device is installed based on an overlapping area of ​​the reading range of the reading device at each of the plurality of device positions. (Configuration 12) the first read data includes information indicating radio wave intensity from the first wireless device when the reading device reads the first wireless device; The server device according to configuration 11, wherein the second determination means determines the object position of the object in which the same first wireless device is installed based on the distance to the same first wireless device corresponding to the radio wave strength and the overlapping area. (Configuration 13) 13. The server apparatus according to any one of configurations 1 to 12, wherein the reading device and the photographing device are implemented in a same device. (Configuration 14) 12. The server apparatus according to claim 1, wherein the reading device and the photographing device are different devices. (Configuration 15) The server apparatus according to configuration 14, wherein the reading device is carried by a person who operates the photographing device, or is attached to the same mobile body as the photographing device. (Configuration 16) 16. The server apparatus according to any one of configurations 1 to 15, wherein the first wireless device and the second wireless device are RFID tags. (Configuration 17) 17. The server device according to any one of configurations 1 to 16, wherein the server device is configured with a plurality of devices capable of communicating with each other via a network. (Configuration 18) A computer program that, when executed on one or more devices, each having one or more processors, causes the one or more devices to function as a server device according to any one of configurations 1 to 17. (Configuration 19) A management system for managing a plurality of objects, each of which is provided with a first wireless device, comprising: an acquiring means for acquiring, from the reading device, first read data indicating a first read time when the reading device reads the first wireless device, and second read data indicating a second read time when the reading device reads a second wireless device disposed at a predetermined position; a first determination means for repeatedly acquiring a movement amount of the reading device from the reading device, and determining a time history of a device position of the reading device based on the movement amount of the reading device at the second reading time indicated by the second reading data and the predetermined position where the second wireless device is placed; a second determination means for determining an object position of an object in which the first wireless device is provided, based on the device position of the reading device at the first reading time of the first wireless device indicated by the first reading data; a display unit for displaying an image captured by a photographing device associated with the reading device; a third determination means for executing a determination process of determining a shooting range of the captured image based on the device position at the shooting time of the captured image displayed by the display means and the shooting direction of the captured image by the shooting device, and determining that the object position is included in the shooting range; Equipped with The display means displays candidate objects that are all or part of the object whose object position is included in the shooting range. (Configuration 20) 20. The management system according to claim 19, further comprising a recording means for recording a created object selected from the candidate objects displayed on the display means and the captured image displayed on the display means in association with each other.

[0095] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0096] 292: Data management unit, 293: UI control unit

Claims

1. A server apparatus of a management system that manages a plurality of objects, each of which is provided with a first wireless device, comprising: an acquiring means for acquiring, from the reading device, first read data indicating a first read time at which the reading device reads the first wireless device, and second read data indicating a second read time at which the reading device reads a second wireless device disposed at a predetermined position; a first determination means for repeatedly acquiring a movement amount of the reading device from the reading device, and determining a time history of a device position of the reading device based on the movement amount of the reading device at the second reading time indicated by the second reading data and the predetermined position where the second wireless device is placed; a second determination means for determining an object position of an object in which the first wireless device is provided, based on the device position of the reading device at the first reading time of the first wireless device indicated by the first read data; a third determination means for executing a determination process of determining a photographing range of a photographed image based on a device position at a photographing time of the photographed image photographed by the photographing device associated with the reading device and a photographing direction of the photographed image by the photographing device, and determining the object position of the object included in the photographing range; a display means for displaying, on a device that has requested the execution of the determination process, candidate objects that are all or part of the object whose object position is included in the shooting range; A server device comprising:

2. the device that has requested the execution of the determination process is the photographing device, the captured image is an image currently being captured by the photographing device, the photographing time is the current time, The server apparatus according to claim 1 , wherein the third determination means acquires the image capturing direction from the image capturing device.

3. the device that has requested the execution of the determination process is the photographing device, The captured image is an image of image data captured by the imaging device and stored in the imaging device, The server apparatus according to claim 1 , wherein the third determination means acquires the shooting direction and the shooting time from the shooting device.

4. the device that has requested the execution of the determination process is a user device different from the photographing device, the captured image is an image represented by image data captured by the imaging device and stored in the server device or another server device, The server device according to claim 1 , wherein the third determination means acquires the photographing direction and the photographing time from the image data stored in the server device or the separate server device.

5. The display means includes: displaying, on the user device, one or more first wireless devices read by the reading device during a time period designated by the user device; The server device of claim 4, wherein when at least one first wireless device among the one or more first wireless devices is selected by the user device, a display process is performed to display on the user device an image of the first reading time during the time period when the at least one first wireless device was read by the reading device.

6. The server device according to claim 5, wherein the display means performs a process of transmitting the image data for the time period stored in the server device or the other server device to the user device, and performs the display process based on the image data for the time period.

7. The image data stored in the server device or the other server device is associated with information indicating a shooting location, The display means includes: displaying, on the user device, one or more first wireless devices read by the reading device during a time period and at a location specified by the user device; The server device of claim 4, wherein when at least one first wireless device among the one or more first wireless devices is selected by the user device, a display process is performed to display on the user device an image of the first reading time during the time period when the at least one first wireless device was read by the reading device.

8. The server device according to claim 7, wherein the display means performs a process of transmitting the image data for the time period associated with the location, which is stored in the server device or another server device, to the user device, and performs the display process based on the image data for the time period associated with the location.

9. The server device according to claim 1, wherein the third determination means obtains information indicating a determination distance from the device that requested the execution of the determination process, and determines, as the candidate object, an object whose object position is included in the shooting range and whose difference between the object position and the determination distance is within a predetermined value.

10. the first read data includes information indicating radio wave intensity from the first wireless device when the reading device reads the first wireless device; The server device according to claim 1 , wherein the third determination means determines a predetermined number of the objects as the candidate objects in descending order of the radio wave strength from the first wireless device provided on the object whose object position is included in the shooting range.

11. The server device according to claim 1, wherein the second determination means determines each of a plurality of device positions at different first read times based on the first read data indicating that the reading device read the same first wireless device at different first read times, and determines the object position of the object in which the same first wireless device is provided based on an overlapping area of ​​the reading range of the reading device at each of the plurality of device positions.

12. the first read data includes information indicating radio wave intensity from the first wireless device when the reading device reads the first wireless device; The server device according to claim 11 , wherein the second determination means determines the object position of the object in which the same first wireless device is provided based on the distance to the same first wireless device corresponding to the radio wave intensity and the overlapping area.

13. The server apparatus according to claim 1 , wherein the reading device and the photographing device are implemented in a same device.

14. The server apparatus according to claim 1 , wherein the reading device and the photographing device are different devices.

15. The server apparatus according to claim 14 , wherein the reading device is carried by a person who operates the photographing device, or is attached to the same mobile body as the mobile body to which the photographing device is attached.

16. The server apparatus according to claim 1 , wherein the first wireless device and the second wireless device are RFID tags.

17. 2. The server device according to claim 1, wherein the server device is composed of a plurality of devices capable of communicating with each other via a network.

18. A computer program which, when executed on one or more devices, each having one or more processors, causes the one or more devices to function as a server device according to any one of claims 1 to 17.

19. A management system for managing a plurality of objects, each of which is provided with a first wireless device, comprising: an acquiring means for acquiring, from the reading device, first read data indicating a first read time at which the reading device reads the first wireless device, and second read data indicating a second read time at which the reading device reads a second wireless device disposed at a predetermined position; a first determination means for repeatedly acquiring a movement amount of the reading device from the reading device, and determining a time history of a device position of the reading device based on the movement amount of the reading device at the second reading time indicated by the second reading data and the predetermined position where the second wireless device is placed; a second determination means for determining an object position of an object in which the first wireless device is provided, based on the device position of the reading device at the first reading time of the first wireless device indicated by the first read data; a display unit for displaying an image captured by a photographing device associated with the reading device; a third determination means for executing a determination process of determining a shooting range of the captured image based on the device position at the shooting time of the captured image displayed by the display means and the shooting direction of the captured image by the shooting device, and determining that the object position is included in the shooting range; Equipped with The display means displays candidate objects that are all or part of the object whose object position is included in the shooting range.

20. 20. The management system according to claim 19, further comprising a recording means for recording a created object selected from the candidate objects displayed on the display means and the captured image displayed on the display means in association with each other.

Images