Article management device, article management method, data structure for article management data, and program

Abstract

The invention relates to an article management device, an article management method, a data structure for article management data, and a program. An article management device (1200) derives a three-dimensional position of a display region based on a template (label) on the basis of a first captured image and a position of a first imaging device that has captured the first captured image, and stores information indicating the three-dimensional position in a storage medium in association with identification information of the thick plate.

Description

Technical Field

[0001] The present invention relates to an article management device, an article management method, a data structure of article management data, and a program, and is particularly suitable for managing articles. Background Art

[0002] Manage the location of items such as steel products, including thick plates.

[0003] In the technology described in Patent Document 1, the position of the steel plate is managed by registering the XY coordinates of the overhead crane when unloading the steel plate together with the steel plate number of the steel plate.

[0004] In addition, in the technology described in Patent Document 2, based on the movement of the crane in the horizontal and vertical directions and the load detection and other actions, the fixed place of the steel represented by the three-dimensional coordinates is calculated by a fixed place calculation program, and the fixed place information representing the calculated fixed place is stored in correspondence with the steel identification information of the steel.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent No. 5109451

[0008] Patent Document 2: Japanese Patent No. 3734419 Summary of the invention

[0009] Problems to be solved by the invention

[0010] However, in the technology described in Patent Document 1, the information used for steel plate position management is the XY coordinates of the bridge crane. Therefore, the position of the item can only be managed within the range that can be transported by the bridge crane. In addition, in the technology described in Patent Document 1, since the Z coordinate cannot be determined from the XY coordinates of the bridge crane, it is impossible to accurately determine the multiple steel plates when multiple steel plates are stacked, for example.

[0011] In addition, in the technology described in Patent Document 2, the three-dimensional coordinates of the steel material are calculated. However, as in Patent Document 1, the position of the item can only be managed within the range that can be transported by the crane. In addition, the three-dimensional coordinates of the steel material calculated by the technology described in Patent Document 2 are the three-dimensional coordinates of a fixed place calculated by a fixed place calculation program. Therefore, in the technology described in Patent Document 2, the position of the steel material can only be managed in a fixed place predetermined by a fixed place calculation program.

[0012] In actual manufacturing sites, products are sometimes transported by various conveying devices. In addition, products are sometimes transported by operators without using conveying devices. In addition, products are sometimes placed in a storage site where the address of the storage site is not determined. Furthermore, there are also various methods for placing products. In the technologies described in patent documents 1 and 2, it is not easy to manage the position of such products. These problems also exist in items other than products.

[0013] The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to appropriately manage the positions of articles.

[0014] Means for solving problems

[0015] The item management device of the present invention manages the location of items, and is characterized in that it has: one or more image acquisition units for acquiring a captured image of an area including display information displayed on the above-mentioned item; an identification information acquisition unit for acquiring identification information of the above-mentioned item; a position derivation unit for deriving the three-dimensional position displayed by the above-mentioned display information, i.e., the information display position, based on the above-mentioned captured image acquired by the above-mentioned image acquisition unit; and a storage unit for storing the above-mentioned information display position on the above-mentioned item derived by the above-mentioned position derivation unit in a storage medium in association with the identification information of the item acquired by the above-mentioned identification information acquisition unit.

[0016] The item management method of the present invention is to manage the location of items, and is characterized in that it includes the following steps: one or more image acquisition steps, acquiring a captured image of an area including display information displayed on the above-mentioned item; an identification information acquisition step, acquiring identification information of the above-mentioned item; a position derivation step, deriving the three-dimensional position displayed by the above-mentioned display information, that is, the information display position, based on the above-mentioned captured image obtained by the above-mentioned image acquisition step; and a storage step, storing the above-mentioned information display position in the above-mentioned item derived by the above-mentioned position derivation step in a storage medium in association with the identification information of the item obtained by the above-mentioned identification information acquisition step.

[0017] The data structure of the item management data of the present invention is to have display information displayed on the item and non-display information not displayed on the above-mentioned item, and the above-mentioned non-display information includes the three-dimensional position displayed by the above-mentioned display information, that is, the information display position, and the item management device is used to derive the above-mentioned information display position in the item based on a captured image of the area including the display information displayed on the above-mentioned item.

[0018] The program of the present invention is used to enable a computer to execute processing for managing the location of an object, and is characterized in that the program enables the computer to execute the following programs: one or more image acquisition steps, acquiring a captured image of an area including display information displayed on the object; an identification information acquisition step, acquiring identification information of the object; a position derivation step, deriving the three-dimensional position displayed by the display information, i.e., the information display position, based on the captured image acquired through the image acquisition step; and a storage step, storing the information display position in the object derived through the position derivation step in association with the identification information of the object acquired through the identification information acquisition step in a storage medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram for explaining an example of the flow of products in a factory.

[0020] Figure 2 This is a diagram showing an example of a display area for a thick plate.

[0021] Figure 3A This is a diagram showing an example of display content based on a template.

[0022] Figure 3B This is a diagram showing an example of display content based on tags.

[0023] Figure 4 This is a diagram showing an example of loading and unloading thick plates on a transport ship.

[0024] Figure 5 This is a diagram showing a state in which a thick plate is lifted by a loading and unloading device, as viewed from the extending direction of the beam (y-axis direction).

[0025] Fig. 6A It is a diagram showing a first example of a method for conveying thick plates.

[0026] Figure 6B It is a diagram showing a second example of the method of conveying thick plates.

[0027] Figure 6C It is a diagram showing a third example of the method of conveying thick plates.

[0028] Fig.6D It is a diagram showing a fourth example of the method for conveying thick plates.

[0029] Fig. 6E It is a diagram showing the fifth example of the method of conveying thick plates.

[0030] Figure 7 This is a diagram showing an example of loading and unloading thick plates on a transport vehicle.

[0031] Figure 8 This is a diagram showing an example of a state where thick plates are loaded on a transport vehicle.

[0032] Fig. 9A This is a diagram showing a first example of a situation in which thick plates are placed in a storage yard in a factory.

[0033] Fig. 9B This is a diagram showing a second example of a situation where thick plates are placed in a storage yard in a factory.

[0034] Fig. 9C This is a diagram showing a third example of the situation where thick plates are placed in a storage yard in a factory.

[0035] Fig.10 This is a diagram showing an example of how a thick plate is processed by a production facility.

[0036] Fig.11 The diagram shows an example of an original thick plate and a thick plate after being cut.

[0037] Fig.12 This is a diagram showing an example of the functional structure of the article management device.

[0038] Fig.13 This is a diagram showing a first example of the management table.

[0039] Fig.14 This is a diagram showing an example of the position of a template within an article and the position of components of each display item in a display based on the template.

[0040] Fig.15 This is a diagram showing an example of the position of a tag within an article and the position of components of each display item in a display based on the tag.

[0041] Fig.16A This is a diagram showing a first example of information stored in the management table.

[0042] Fig. 16B It is a diagram showing a second example of information stored in the management table.

[0043] Fig.17 This is a flowchart for explaining a first example of processing of the article management device.

[0044] Fig.18 This is a diagram for explaining an example of a method of deriving a three-dimensional position of a display area based on a template.

[0045] Fig.19 This is a diagram for explaining an example of a method of deriving the position of the display area in the height direction (z-axis direction) based on the template.

[0046] Fig. 20AThis is a diagram showing an example of a first captured image including a display area based on a template.

[0047] Fig. 20B This is a diagram showing an example of the geometric relationship between the interior of the first imaging device and the display area based on the template.

[0048] Fig.21 This is a diagram for explaining a modified example of a method of deriving a three-dimensional position of a display area based on a template.

[0049] Fig. 22 This is a diagram showing an example of a steel frame of a structure installed on the ground.

[0050] Fig.23 This is a diagram showing an example of a pipe installed underground.

[0051] Fig.24 This is a diagram showing a second example of the management table.

[0052] Fig.25 This is a flowchart for explaining the second example of the processing of the article management device.

[0053] Fig.26 This is a diagram showing an example of the hardware configuration of the article management device. DETAILED DESCRIPTION

[0054] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0055] Furthermore, when the lengths, positions, sizes, intervals, etc., etc., are the same, in addition to being strictly the same, they also include being different within the scope of the invention (for example, being different within the tolerance range determined during design).

[0056] (First embodiment)

[0057] First, the first embodiment is described. In this embodiment, an example is given of managing the position of an object whose position changes over time or whose placement changes over time. Such position management is also called tracking. Tracking means tracking and determining the position of an object. In this embodiment, an example is given of managing the position of various thick plates (e.g., thick steel plates) manufactured by multiple iron and steel plants in a factory that performs various processes such as heat treatment and cutting. Therefore, first, an overview of the factory to which this embodiment is applied is described.

[0058] <Factory Overview>

[0059] Figure 1 This is a diagram for explaining an example of the flow of products in a factory to which the present embodiment is applied.

[0060] exist Figure 1 In the process of transporting thick plates produced by the ironworks, the thick plates are loaded onto transport ships. After that, the transport ships dock at the pier near the factory. The thick plates are unloaded from the transport ships and loaded onto transport vehicles (trucks, etc.). When the thick plates transported by the transport vehicles arrive at the factory, they are placed in the storage yard inside the factory. After the thick plates placed in the storage yard are subjected to heat treatment, cutting (gas cutting), etc., they are placed in the warehouse. After that, the thick plates are shipped from the factory to the demander.

[0061] <Concept>

[0062] When the position of the thick plate transported as described above is managed based on the action of the conveying equipment as described in Patent Documents 1 and 2, the range of the manageable position is limited. Therefore, the inventors of the present invention focus on the situation where information managed by the manufacturer of the thick plate is displayed on the thick plate for the purpose of spot management (management of the position of each thick plate). Managing the display position of such information is equivalent to managing the position of the thick plate. Then, only by being able to determine such information managed by the manufacturer of the thick plate, the thick plate can be identified. The present embodiment is completed based on this idea.

[0063] <Display for thick plates>

[0064] Figure 2 is a diagram showing an example of a display area for a thick plate. Figure 2 The x-axis, y-axis, and z-axis shown represent Figure 2 and Figure 3A , Figure 3B , Figure 14-15 , Figure 18-19 , Fig.21 The direction of the relationship between Figure 4 as well as Figure 5 The xyz coordinates correspond to the directions of the xyz coordinates shown. In addition, the xyz coordinates are coordinates represented by the world coordinate system, for example.

[0065] exist Figure 2 In FIG. 2 , display area 201 is a display area based on a template. Figure 2 In the example shown, the template-based display area 201 exists on the surface of the thick board. Figure 2 The dashed lines shown are imaginary lines and are not actually shown. Figure 2 In the example shown, the label (side label) 202 is attached to the side surface (plate thickness portion) of the thick plate.

[0066] Figure 3A is a diagram showing an example of display content based on a template. In addition, as described above, Figure 3A The xyz coordinates shown are Figure 2 In addition, Figure 3A In the xyz coordinates shown, a symbol in which a black circle (●) is added to a white circle (○) indicates that the symbol is from the back side to the front side of the paper. This xyz coordinate notation is the same in other figures.

[0067] exist Figure 3A In the example shown, the display based on the template includes a tag 301 , a buyer name 302 , a specification 303 , a size 304 , an ID 305 , a buyer code 306 , and an order number 307 .

[0068] The mark 301 is a mark indicating the manufacturer of the thick plate. The buyer name 302 is information indicating the buyer (buyer) of the thick plate. The specification 303 is information indicating the specification of the thick plate. The size 304 is information indicating the size of the thick plate (for example, thickness × width × length). The ID 305 is information for uniquely identifying the thick plate. Figure 3A In the example shown, it is a plate number (identification number of a thick plate). Therefore, the same ID is not added to different thick plates. The customer code 306 is information assigned to the thick plate by the customer by the manufacturer of the thick plate. The order number 307 is a part of the number used to identify the order of the thick plate from the customer.

[0069] The mark 301, the customer name 302, the specification 303, the size 304, the ID 305, the customer code 306 and the order number 307 are information that can be generally displayed for thick plates. In addition, information other than the ID 305 may not be included in the display based on the template. In addition, information other than the above information may also be included in the display based on the template.

[0070] In addition, for example, the number of digits of the demander code 306 may differ depending on the demander, or the method of expressing the specification 303 may differ depending on the ironworks that manufactures the thick plate. In addition, for example, the order number 307 includes a case where it is added by the demander and a case where it is not added. In addition, the size of the display item may also differ depending on the ironworks that manufactures the thick plate. In addition, the position of the display area based on the template may also differ depending on the ironworks that manufactures the thick plate. In this way, the content, position, and size of the display based on the template may differ depending on the ironworks that manufactures the thick plate and the demander, etc.

[0071] Mark 301, demander name 302, specification 303, size 304, ID 305, demander code 306, and order number 307 are information managed by the manufacturer of the thick plate. In addition, mark 301, demander name 302, specification 303, size 304, ID 305, demander code 306, and order number 307 are information that can be recognized by humans. Information that can be recognized by humans means that people can understand the meaning of the information when they see the information. Information that needs to be encoded and decoded by machines when the information is produced and read, such as barcodes and QR codes, is not information that can be recognized by humans. Specifically, mark 301, demander name 302, specification 303, size 304, ID 305, demander code 306, and order number 307 are information that does not need to be decoded by machines when reading, and include at least one of text, numbers, and symbols, and do not include information that needs to be decoded by machines when reading.

[0072] Figure 3B is a diagram showing an example of display content based on tags. In addition, as described above, Figure 3B The xyz coordinates shown are Figure 2 In addition, Figure 3B In the xyz coordinates shown, a symbol with an x ​​(cross mark) added to a white circle indicates a direction from the front side to the back side of the paper surface. Such xyz coordinate notation is the same in other figures.

[0073] exist Figure 3B In the example shown, the tag-based display includes specification 311, ID 312, size 313, delivery date 314, and demander code 315. Delivery date 314 is information indicating the delivery date of the thick plate. Specification 311, ID 312, size 313, and demander code 315 are respectively the same as specification 303, size 304, ID 305, and demander code 306. In addition, information other than ID 312 may not be included in the tag-based display. In addition, information other than the above information may also be included in the tag-based display. Similar to the template-based display, the content, position, and size of the tag-based display also vary depending on the steelworks that manufactures the thick plate and the demander, etc.

[0074] However, the content represented by the same display item for the same slab is the same in both the template-based display and the label-based display. Figure 3A The template-based display and Figure 3BIn the case of the tag-based display shown, the ID305 displayed by the template and the ID312 displayed by the tag represent the same content. In addition, the specification 303 displayed by the template and the specification 311 displayed by the tag represent the same content. The size 313 and the demander code 315 also represent the same content as the size 304 and the demander code 306, respectively. As described above, the specification 303, ID305, the size 304 and the demander code 306 are information managed by the manufacturer of the thick plate. Therefore, the specification 311, ID312, the size 313 and the demander code 315 are also information managed by the manufacturer of the thick plate. In addition, the delivery date 314 of the thick plate is also information managed by the manufacturer of the thick plate. In addition, the specification 311, ID312, the size 313, the delivery date 314 and the demander code 315 are information that can be recognized by humans.

[0075] In addition, for a thick plate, in addition to the above-mentioned display, for example, a display by engraving may be implemented. In addition, a display by handwriting may be implemented for a thick plate.

[0076] <Conveying method for thick plates>

[0077] <<Transportation by transport ship>>

[0078] Figure 4 FIG. 1 is a diagram showing an example of loading and unloading thick plates on a transport ship. Figure 4 , a case where a thick plate is loaded and unloaded with respect to a transport ship 400 using a gantry crane 410 is illustrated. Figure 5 This is a diagram showing a state in which the thick plate 420 is lifted by the loading and unloading device 411 as viewed from the extending direction (y-axis direction) of the beam 412 . Figure 4 as well as Figure 5 The xyz coordinates shown represent Figure 4 as well as Figure 5 direction relationship.

[0079] The gantry crane 410 has a loading and unloading device (trolley, etc.) 411, a beam (horizontal frame) 412, and a leg structure 413. The loading and unloading device 411 moves along the extending direction (y-axis direction) of the beam 412 while lifting the object to be transported. Figure 4 , the thick plate 420 is shown to be lifted and moved by the loading and unloading device 411. When the thick plate 420 is lifted by the loading and unloading device 411 from the extending direction of the crossbeam 412 (y-axis direction), as shown in FIG. Figure 5 shown.

[0080] In addition, Figure 4 as well as Figure 5In FIG. 4 , a case where a thick plate 420 is hoisted by the handling device 411 is illustrated. However, the method of conveying the thick plate by the handling device 411 is not limited to this method. Figure 6A to Figure 6E The following are diagrams showing the first to fifth examples of thick plate conveying methods. Figure 6A to Figure 6E In FIG. 4 , the loading and unloading device 411 is shown in a more simplified manner.

[0081] exist Fig. 6A In FIG. 4 , two thick plates are shown being transported simultaneously by the loading and unloading device 411 while being arranged in the horizontal plane (xy plane). Figure 6B , a state where a thick plate is conveyed on the top side (negative direction side of the x-axis) of the loading and unloading device 411 is illustrated. Figure 6C In FIG. 4 , a thick plate is conveyed on the bottom side (positive direction side of the x-axis) of the loading and unloading device 411. Fig.6D In FIG. 4 , a thick plate is transported in the center of the loading and unloading device 411. Fig. 6E , a state is shown where a plurality of (two) thick plates are simultaneously transported in a stacked state by the loading and unloading device 411. Thus, in this embodiment, a plurality of thick plates may be simultaneously transported in a stacked state.

[0082] As described above, there is no limitation on the conveying method of the thick plate by the handling device 411. In addition, the method of lifting the thick plate by the handling device 411 may be a method using the magnetic attraction of a magnet, a method using a hook, or other methods.

[0083] The gantry crane 410 itself can be described using a known technique, and therefore a detailed description thereof will be omitted here.

[0084] The thick plates manufactured by the ironworks are transported to the dock and placed in a designated storage area where they can be transported by the gantry crane 410. Figure 4 As shown in FIG. 1 , a plurality of thick plates are sometimes placed in a stacked state at a predetermined storage area. Sometimes, the thick plates are placed one by one at a predetermined storage area. Figure 4 In the examples, for the sake of convenience, the case where the thickness (plate thickness) of multiple thick plates is the same is illustrated. However, the thickness of multiple thick plates may be different. That is, sometimes at least one of the thickness, width and length of the thick plates placed in a specified placement area is different.

[0085] exist Figure 4 In the example, a case where a plurality of thick plates are placed in a stacked manner in such a manner that the plate surface of the thick plates is parallel to the surface on which the thick plates are placed is illustrated. However, the method of placing the thick plates in a stacked manner is not limited to this method. For example, sometimes a plurality of thick plates are placed in a stacked manner in such a manner that the plate surface of the thick plates is perpendicular or nearly perpendicular to the surface on which the thick plates are placed. This situation is also the same at locations other than the specified storage area of ​​the dock.

[0086] In the present embodiment, the first camera devices 430a to 430f are installed on the beam 412 and the leg structure 413. The number and position of the first camera devices 430a to 430f are set so that as many display areas of the thick plate as possible can be photographed as the template-based display area 201 and the label-based display area 202 of the thick plate. In addition, the first camera devices 430g to 430m are also installed on the loading and unloading device 411. The number and position of the first camera devices 430a to 430m are set so that at least one of the template-based display area 201 and the label-based display area 202 of the thick plate 420 being transported by the gantry crane 410 can be photographed as much as possible.

[0087] The first camera devices 430a to 430m can adjust the shooting conditions (shooting direction, field of view angle, zoom ratio, aperture (F value), etc.). The adjustment of the shooting conditions can be performed by an operator (worker) operating the first camera device, or can be performed automatically based on instructions from an external device. In addition, for example, the first camera devices 430i to 430m installed at a position close to the thick plate 420 being transported among the first camera devices 430g to 430m installed on the beam 412 can also be fiber optic cameras. If a fiber optic camera is used, for example, the template-based display area 201 and the label-based display area 202 of the thick plate 420 being transported can be photographed at close range.

[0088] In addition, the image captured by the first camera is not limited to a still image, but may be a dynamic image. In the case where the image captured by the first camera is a dynamic image, an image taken from the dynamic image captured by the first camera (i.e., a frame image of the dynamic image) is used in the item management device 1200 described later. In this case, it is preferable to take out a still image that is as focused as possible on the subject captured by the first camera from the dynamic image captured by the first camera. The case where the image captured by the camera is not limited to a still image also applies to the second to seventh cameras described later.

[0089] The thick plate transported from the ironworks is placed at the designated storage site of the gantry crane 410. The area including the display area 201 based on the template of the thick plate placed at the designated storage site of the gantry crane 410 is photographed by the first imaging devices 430c to 430f. The operator (worker) operates the information processing terminal located in the operation room of the gantry crane 410 or the information processing terminal (tablet terminal, etc.) held by the operator (worker) of the gantry crane 410 to instruct the first imaging devices 430c to 430f to display the preview image. The first imaging devices 430c to 430f send the preview image to the information processing terminal based on the display instruction of the preview image. Then, the preview image is displayed by the information processing terminal. The operator (worker) operates the information processing terminal to instruct the first imaging device to photograph and display the preview image of the display area based on the template to be photographed. When receiving the photographing instruction, the first imaging devices 430c to 430f photograph the image of the area including the display area 201 based on the template. In addition, the preview image is a real-time image displayed for confirmation by the operator (worker), and is displayed based on the display instruction of the preview image. The captured image is an image captured based on the capture instruction when the preview image is displayed. In addition, the case of displaying the preview image is illustrated here, but the preview image does not necessarily have to be displayed.

[0090] When a plurality of thick plates are stacked and conveyed simultaneously, the area including the display area 201 based on the template cannot be photographed for the thick plate located at the bottom. In this case, instead of the photographed image of the area including the display area 201 based on the template, the photographed image of the area including the display area based on the label 202 is photographed.

[0091] Depending on the position where the thick plate is placed, the display area 201 based on the template and the display area based on the label 202 are sometimes not included in the shooting range of the first camera 430c~430f. In this case, the operator (operator) can also operate the unmanned aerial vehicle (drone) 440a~440b equipped with the first camera, and use the first camera to shoot the image of the area including the display area 201 based on the template (or the area including the display area based on the label 202) of the desired thick plate. In addition, the image of the area including the display area 201 based on the template (or the area including the display area based on the label 202) of the desired thick plate can also be shot by the first camera held by the operator (operator). In addition, the first camera held by the operator (operator) can also be, for example, an information terminal device (tablet terminal, smart phone, etc.) equipped with the first camera.

[0092] In addition, the display of the preview image and the shooting of the shooting image of the above-mentioned first camera devices 430c~430f are also performed in the same manner in the first camera devices 430a~430b, 430g~430w other than the first camera devices 430c~430f. In addition, in the case where a plurality of thick plates are stacked and conveyed simultaneously, the case where the shooting image of the area including the display area based on the label 202 is shot is also performed in the same manner in the first camera devices 430a~430b, 430g~430w other than the first camera devices 430c~430f. In addition, the shooting method in the case where the display area 201 based on the template and the display area based on the label 202 are not included in the shooting range of the first camera devices 430c~430f is also the same for the first camera devices 430a~430b, 430g~430w other than the first camera devices 430c~430f. Therefore, in the following, detailed descriptions of the first imaging devices 430a to 430b and 430g to 430w other than the first imaging devices 430c to 430f will be omitted.

[0093] After the thick plates are placed at a predetermined placement site of the gantry crane 410, they are loaded onto the transport ship 400 by the gantry crane 410. Sometimes, a plurality of thick plates are loaded onto the transport ship 400 in a stacked state at the same time. Sometimes, the thick plates are loaded onto the transport ship 400 one by one. In addition, sometimes, at least one of the thickness, width, and length of the thick plates loaded onto the transport ship 400 is different.

[0094] The first imaging devices 430 g to 430 m installed in the loading and unloading device 411 capture images of a region including the display region 201 based on the template of the thick plate being conveyed by the gantry crane 410 .

[0095] When a thick plate is loaded onto the transport ship 400, a first camera 430a-430b installed on the beam 412 or a first camera 430n installed on the transport ship 400 captures an image of the area of ​​the thick plate including the display area 201 based on the template. Similar to the first camera 430a-430m, the first camera 430n installed on the transport ship 400 can also adjust the shooting conditions (shooting direction, field of view, zoom ratio, aperture (F value), etc.). Figure 4 In the figure, for convenience of description, the number of the first imaging device installed in the transport ship 400 is set to one. However, on the transport ship 400, a plurality of first imaging devices are installed so as to include the region where the thick plate is placed in the imaging range.

[0096] The transport ship 400 loaded with thick plates as described above docks at a pier near the factory. The thick plates are unloaded from the transport ship 400 as described above. As described above, in the present embodiment, the thick plates are unloaded using the gantry crane 410. For the thick plates being transported by the gantry crane 410, images are captured in the same manner as when they are loaded onto the transport ship 400. Furthermore, when the thick plates are unloaded by the gantry crane 410 and placed at a predetermined storage area of ​​the gantry crane 410, images are captured in the same manner as when they are transported from the ironworks and placed at a predetermined storage area of ​​the gantry crane 410.

[0097] <<Transportation based on transport vehicles>>

[0098] The thick plates placed at a prescribed placement site of the gantry crane 410 are loaded onto a transport vehicle. Figure 7 4 is a diagram showing an example of loading or unloading thick plates on a transport vehicle. The thick plates placed at a predetermined storage area of ​​the gantry crane 410 are loaded onto the transport vehicle 700 using a crane (not shown). Figure 7 As shown in FIG. 1 , sometimes a plurality of thick plates are simultaneously loaded onto the transport vehicle 700 in a stacked state. However, sometimes the thick plates are loaded onto the transport vehicle 700 one by one. In addition, sometimes at least one of the thickness, width, and length of the thick plates loaded onto the transport vehicle 700 is different. Figure 6A to Figure 6E The thick plates are transported and loaded onto the transport vehicle 700 as shown.

[0099] Figure 8 1 is a diagram showing an example of a state in which thick plates are loaded on a transport vehicle 700. The first camera devices 430o to 430p are installed on a structure set at a standby position of the transport vehicle 700. The number and positions of the first camera devices 430o to 430p are set so as to capture as many display areas of the thick plates as possible as the template-based display area 201 and the label-based display area 202 of the thick plates loaded on the transport vehicle 700. In addition, the first camera device 430q is also installed on the transport vehicle 700. Similar to the first camera devices 430a to 430m, the first camera devices 430o to 430q can also adjust the shooting conditions (shooting direction, field of view angle, zoom ratio, aperture (F value), etc.).

[0100] The first imaging devices 430o to 430q capture images of the area including the template-based display area 201 of the thick plate loaded on the transport vehicle 700.

[0101] <<Transportation within the factory>>

[0102] When the transport vehicle 700 loaded with the thick plates as described above arrives at the factory, the thick plates are unloaded from the transport vehicle 700. The thick plates loaded on the transport vehicle 700 are unloaded from the transport vehicle 700 using a crane or the like not shown. Figure 7 As shown in FIG. 1 , sometimes a plurality of thick plates are unloaded from the transport vehicle 700 in a stacked state at the same time. However, sometimes the thick plates are unloaded from the transport vehicle 700 one by one. In addition, sometimes at least one of the thickness, width and length of the thick plates unloaded from the transport vehicle 700 is different. In addition, it is also possible to Figure 6A to Figure 6E The thick plates are transported as shown and unloaded from the transport vehicle 700. In addition, the conveying equipment (gantry crane 410) used when conveying the thick plates from the transport ship 400 to the designated storage yard of the gantry crane 410 is different from the conveying equipment (crane etc. not shown) used when conveying the thick plates from the transport vehicle 700 to the storage yard in the factory. In this way, the thick plates transported to a storage yard by a certain conveying equipment are sometimes transported to other storage yards that cannot be transported by the certain conveying equipment by a conveying equipment different from the certain conveying equipment. The same is true when the thick plates are transported between other storage yards such as between storage yards in the factory.

[0103] Figures 9A to 9C 1 to 3 are diagrams showing the first to third examples of thick plates placed in a storage yard in a factory. First imaging devices 430r to 430w are installed on the structures set up in the storage yard in the factory. Figures 9A to 9C As shown, the thick plates are placed in the storage field in the factory in various ways. Therefore, the number and position of the first camera devices 430r~430w are set to capture as many display areas of the thick plates as possible as the display areas 201 based on the template and the display areas based on the labels 202 of the thick plates placed in the storage field. Similar to the first camera devices 430a~430m, the first camera devices 430r~430w can also adjust the shooting conditions (shooting direction, field of view angle, zoom ratio, aperture (F value), etc.).

[0104] The first imaging devices 430r to 430w capture images of a region including the display region 201 based on the template, of the thick plates placed in the storage yard within the factory.

[0105] When a thick plate is placed in the loading site, as described above, the first imaging devices 430r to 430w capture images of the area of ​​the thick plate including the display area 201 based on the template (or the display area based on the label 202).

[0106] Afterwards, the slabs are transported to the production facility. Fig.10 This is a diagram showing an example of a thick plate being processed by a production facility. Fig.10In the example, the production equipment 1000 is a device for heat treatment and a device for gas cutting. Fig.10 In the example, the thick plates 1002a and 1002b are transported by the transport rollers 1001. The thick plates are transported between the storage area and the transport rollers 1001 by using a transport device (not shown).

[0107] The first camera devices 430x to 430y are installed on a structure of a production line set up in a factory. The number and position of the first camera devices 430x to 430y are set so as to be able to capture the display area 201 based on the template of the thick plates 1002 and 1002b. Like the first camera devices 430a to 430m, the first camera devices 430x to 430y can also adjust the shooting conditions (shooting direction, field of view angle, zoom ratio, aperture (F value), etc.). In addition, the thick plates 1002a and 1002b transported on the conveying roller 1001 will not overlap. In addition, the conveying path of the thick plates 1002a and 1002b on the conveying roller 1001 is pre-assumed. Therefore, in this embodiment, the display area based on the label 202 of the thick plates 1002a and 1002b will not be missed by the first camera devices 430x to 430y. However, an unmanned aerial vehicle equipped with a first imaging device may be used to image the display area of ​​the thick plates 1002a and 1002b based on the tag 202. Fig.10 In the figure, the dotted lines indicate the conveying paths of the thick plates 1002a and 1002b on the conveying rollers 1001, and do not actually exist.

[0108] The thick plate 1002a transported from the storage area in the factory is placed on the transport roller 1001 and transported by the transport roller 1001. The area including the display area 201 based on the template of the thick plate 1002a being transported on the transport roller 1001 before entering the production equipment 1000 is photographed by the first imaging device 430x. In addition, after the processing in the production equipment 1000 is completed and the thick plate 1002b is discharged from the production equipment 1000, the area including the display area 201 based on the template of the thick plate 1002b being transported on the transport roller 1001 is photographed by the first imaging device 430y.

[0109] Afterwards, the slabs are placed in the warehouse Figures 9A to 9C As described above, the area including the display area 201 based on the template (or the display area based on the label 202) of the thick plate placed in the storage area in the factory is photographed by the first camera 430r~430w. When shipping, the thick plate is moved from the warehouse to the storage area for shipping. The area including the display area 201 based on the template (or the display area based on the label 202) of the thick plate placed in the storage area for shipping is photographed by the first camera 430r~430w.

[0110] In each storage area in the factory, a plurality of thick plates are sometimes placed in a stacked state. In other cases, thick plates are placed one by one in each storage area in the factory. In addition, at least one of the thickness, width and length of the thick plates placed in each storage area in the factory is sometimes different.

[0111] Fig.11 This is a diagram showing an example of an original thick plate and a thick plate after cutting. Fig.11 In FIG. 1 , a case where a thick plate 1100 is cut to produce thick plates 1110, 1120, 1130, and 1140 is illustrated. After the thick plates 1110, 1120, 1130, and 1140 are produced in this way, each of the thick plates 1110, 1120, 1130, and 1140 is displayed. Fig.11 , display areas 1111, 1121, 1131, 1141 for the slabs 1110, 1120, 1130, 1140 are shown. Information including the IDs of the slabs 1110, 1120, 1130, 1140 is displayed in the display areas 1111, 1121, 1131, 1141. For example, a display based on a template is performed in the display areas 1111, 1121, 1131, 1141. In addition to the display based on the template, at least one of a display based on an engraving and a display based on a label may be performed. In addition, the display in the display areas 1111, 1121, 1131, 1141 may also be a display based on handwriting.

[0112] In addition, the ID of the original thick plate 1100 may be displayed in the display areas 1111, 1121, 1131, and 1141. For example, the display in the display areas 1111, 1121, 1131, and 1141 is based on a template. However, the display in the display areas 1111, 1121, 1131, and 1141 may be based on handwriting. This is also the case with the display in the display area 201 and the display based on the label 202. In the following, it is assumed that the display areas 1111, 1121, 1131, and 1141 are display areas based on a template.

[0113] When the thick plates 1110, 1120, 1130, 1140 are manufactured, for example, the first camera held by the operator (worker) can be used to capture images of the desired thick plates including the display areas 1111, 1121, 1131, 1141 based on the template. In addition, the first camera held by the operator (worker) can be, for example, an information terminal device (tablet terminal, smart phone, etc.) equipped with the first camera. In addition, the operator (worker) can also operate an unmanned aerial vehicle equipped with the first camera to capture images of the desired thick plates including the display areas 1111, 1121, 1131, 1141 based on the template through the first camera.

[0114] As described above, the present embodiment illustrates the following situation: at least when placed on the transport ship 400, when unloaded from the transport ship 400 and loaded onto the transport vehicle 700, and when placed on the gantry crane 410 and various placement sites of the factory, captured images of each thick plate including areas of the display areas 201, 1111, 1121, 1131, 1141 based on the template and areas of the display area based on the label 202 are obtained.

[0115] In the present embodiment, the position of the thick plate is automatically determined by the article management device using the captured image captured as described above. In addition, when the thick plates are conveyed one by one, the display area 201 based on the template can be captured, so it is not necessary to include the captured image of the area of ​​the display area based on the label 202. In the following description, when the display based on the template and the display based on the label 202 are collectively referred to, they are expressed as a display based on the template (label) as needed. In addition, when the display areas 201, 1111, 1121, 1131, 1141 based on the template and the display area based on the label 202 are collectively referred to, they are expressed as a display area based on the template (label) as needed.

[0116] <Item Management Device>

[0117] Fig.12 This is a diagram showing an example of the functional configuration of the article management device 1200. The hardware of the article management device 1200 is realized by using, for example, an information processing device including a CPU, ROM, RAM, HDD, and various interfaces, or dedicated hardware. Fig.13 1 is a diagram showing an example of a management table 1300 managed by the article management device 1200. In addition, as described above, in this embodiment, tracking is performed by the article management device 1200. Therefore, in this embodiment, the article management device 1200 is a tracking device.

[0118] like Fig.12As shown, in this embodiment, the item management device 1200 has a first image acquisition unit 1201, a recognition unit 1202, an acquisition unit 1203, a comparison unit 1204, a determination unit 1205, a position derivation unit 1206, a storage unit 1207, an output unit 1208 and a second image acquisition unit 1209.

[0119] The first image acquisition unit 1201 acquires a first captured image of an area including a display area based on a template (label). The recognition unit 1202 recognizes the components, positions, etc. of display items in the display based on the template (label) based on the first captured image acquired by the first image acquisition unit 1201. The acquisition unit 1203 acquires information stored in the ID column of the spot display column in the management table 1300 described later. The comparison unit 1204 compares the information recognized by the recognition unit 1202 with the information acquired by the acquisition unit 1203. The determination unit 1205 determines one thick plate from the thick plates managed in the management table 1300 based on the comparison result of the comparison unit 1204. In the present embodiment, the identification information acquisition unit 1210 that acquires the identification information of the thick plate includes the recognition unit 1202, the acquisition unit 1203, the comparison unit 1204, and the determination unit 1205. The position derivation unit 1206 derives the xyz coordinates (three-dimensional coordinates) of the display area based on the template (label) as the three-dimensional position of the thick plate determined by the determination unit 1205. The storage unit 1207 stores the information indicating the three-dimensional position of the thick plate in the management table 1300. The output unit 1208 outputs the information indicating the three-dimensional position of the thick plate. The second image acquisition unit 1209 acquires the second captured image, which is used to determine the position of the display area based on the template (label) in the height direction (z-axis direction) in the position derivation unit 1206. For a detailed example of the item management device 1200, refer to Fig.17 The flowchart is described later.

[0120] exist Fig.13 In the management table 1300, information used by the article management device 1200 to manage the position of the thick plate is stored. The information stored in the management table 1300 includes information managed by the manufacturer of the thick plate. However, a part of the information stored in the management table 1300 may also be information that can be used in circulation, etc. Here, the information managed by the manufacturer of the thick plate refers to information set or obtained by the manufacturer of the thick plate. The information managed by the manufacturer of the thick plate includes information that the employees of the manufacturer of the thick plate, i.e., the operators, can understand the meaning of the information. More specifically, the information managed by the manufacturer of the thick plate is, for example, information produced by the manufacturer of the thick plate and stored in the management table 1300. In the present embodiment, the information stored in the management table 1300 is an example of article management data, and an example of a data structure of article management data is realized by storing each information in the management table 1300.

[0121] exist Fig.13 In the figure, information in the same column is identification information for the same thick plate (columns under the same No. are columns for storing identification information for the same thick plate).

[0122] exist Fig.13 In the present goods display column, information related to each display item displayed based on the template (label) is stored. Information other than information related to each display item displayed based on the template (label) is stored in the non-present goods display column. In this embodiment, the information stored in the present goods display column is an example of display information included in the item management data, and the information stored in the non-present goods display column is an example of non-display information included in the item management data.

[0123] An example of information stored in the on-hand product display column of the management table 1300 will be described.

[0124] In the stock display column, information related to the template is stored in the template column.

[0125] The ID column of the template column stores information indicating the content and position of the component of ID 305 included in the display based on the template. The component refers to each piece of information constituting a display item, such as a character, a number, a symbol, and a mark.

[0126] The customer name column of the template column stores information indicating the contents and positions of components of the customer name 302 included in the display based on the template.

[0127] The size column of the template column stores information indicating the contents and positions of components of the size 304 included in the display based on the template.

[0128] The specification column of the template column stores information indicating the contents and positions of components of the specification 303 included in the display based on the template.

[0129] In the customer code column of the template column, information indicating the contents and positions of components of the customer code 306 included in the display based on the template is stored.

[0130] The order number column of the template column stores information indicating the contents and positions of components of the order number 307 included in the display based on the template.

[0131] In addition to the above display items in the template-based display, the contents of display items that may be displayed as display items in the template-based display are also stored in the management table 1300 . Fig.13 This is indicated by the “…” below the additional information column of the template column. In addition, the display items themselves can also be added and / or deleted.

[0132] In the stock display column, information related to the tag is stored in the tag column.

[0133] The ID column of the tab column stores information indicating the content and position of the component of ID 312 included in the display based on the tab.

[0134] The size column of the tab column stores information indicating the content and position of components of the size 313 included in the display based on the tab.

[0135] The specification column of the tab column stores information indicating the content and position of the components of the specification 311 included in the display based on the tab.

[0136] The delivery date column of the tag column stores information indicating the content and position of the component of the delivery date 314 included in the display based on the tag.

[0137] The customer code column of the tag column stores information indicating the content and position of the components of the customer code 315 included in the display based on the tag.

[0138] In addition to the above display items in the tag-based display, the contents of display items that may be displayed as display items of the tag-based display are also stored in the management table 1300 . Fig.13 This is indicated by the "..." below the demand side code column in the label column. In addition, the display items themselves can also be added and / or deleted.

[0139] Next, an example of information stored in the out-of-stock display column of the management table 1300 will be described.

[0140] The in-article position column stores information indicating the position of the display area when a predetermined position in the thick plate is used as the origin (reference).

[0141] The template column of the in-article position column stores information indicating the position of the display area based on the template when a predetermined position in the thick plate is used as the origin (reference).

[0142] The label column of the in-article position column stores information indicating the position of the display area based on the label when a predetermined position in the thick plate is used as an origin (reference).

[0143] The information indicating the three-dimensional position of the thick plate is stored in the item position column. The information stored in the item position column is information indicating the three-dimensional position of the thick plate determined by the item management device 1200. Therefore, the three-dimensional position of the same thick plate determined by the item management device 1200 changes not only the x-coordinate and the y-coordinate but also the z-coordinate in some cases as the thick plate moves. Therefore, the three-dimensional position of the information display position of the same thick plate determined by the item management device 1200 changes not only the x-coordinate and the y-coordinate (position in the horizontal direction) but also the z-coordinate (position in the height direction) in some cases as the thick plate moves.

[0144] The registration time column stores information indicating the date and time (timing specified by the date and time) when the plank was placed at the current position.

[0145] The main ID column stores information indicating the ID of the original thick plate. Sometimes the thick plate is cut in the factory. The original thick plate refers to the thick plate before cutting. Therefore, when the main ID column stores information indicating the ID of the original thick plate, the information stored in the column in the same column as the main ID column is information about the thick plate after cutting. Fig.11 When thick plates 1110, 1120, 1130, and 1140 are manufactured as shown, in the management table 1300, the content and position of the components of the display items displayed in the display areas 1111, 1121, 1131, and 1141 are stored in the stock display column for the thick plates 1110, 1120, 1130, and 1140. In addition, the information indicating the ID of the thick plate 1100 is stored in the main ID column for the thick plates 1110, 1120, 1130, and 1140. Thus, the following case is exemplified in this embodiment: information about the thick plate after cutting is stored in Fig.13 This is managed as a new column (No) in the management table 1300 shown. In the management table 1300, information on the cut thick plate is stored in the columns of the current stock display and the non-current stock display.

[0146] Here, for example, sometimes the thick plate 1110 is cut and further manufactured into multiple other thick plates. In this case, information indicating the ID of the thick plate 1110 is stored in the main ID column for the multiple other thick plates. Information indicating the ID of the thick plate 1100 is stored in the main ID column for the thick plate 1110. In this way, by tracing the information stored in the main ID column, the changes of each thick plate can be determined. In this way, for a certain thick plate, the arrangement of the thick plates obtained by tracing the information stored in the main ID column is called the cutting plate series of the thick plate. For example, the cutting plate series of the thick plate 1110 is the thick plates 1100 and 1110. In addition, in the main ID column of each thick plate, for example, in order to know the cutting plate series, not only the information indicating the ID of the nearest thick plate is stored, but also the information indicating the ID of all the thick plates belonging to the cutting plate series is stored. In this case, any ID uses the ID defined by the manufacturer.

[0147] The ID of the transport tool of the thick plate is stored in the transport tool ID column. In this embodiment, the case where the transport tool is the transport ship 400 and the transport vehicle 700 is exemplified.

[0148] When the thick plates are loaded on the transport ship 400, the article management device 1200 may store information indicating the ID of the transport ship 400, for example, as follows.

[0149] First, the article management device 1200 obtains the ID of the plate and the ID (so-called ship number) of the transport ship 400. Then, the article management device 1200 identifies the column storing information indicating the same ID as the obtained ID of the plate from the ID column of the management table 1300. Then, the article management device 1200 stores information indicating the obtained ID of the transport ship 400 in the transport tool ID column of the identified column.

[0150] For example, the first imaging devices 430a, 430b, and 430n may capture an image including the display area 201 based on the template or the display area based on the label 202, and an area displaying information indicating the ID of the transport ship 400. In this case, the article management device 1200 may also recognize and obtain the ID of the thick plate and the ID of the transport ship 400 from the captured image.

[0151] Furthermore, the first imaging devices 430a, 430b, and 430n may capture an image including the display area 201 based on the template or the display area based on the tag 202, and a captured image including the area where the information indicating the ID of the transport ship 400 is displayed. For example, after capturing an image including the display area 201 based on the template or the display area based on the tag 202, a captured image including the area where the information indicating the ID of the transport ship 400 is displayed may be captured. In this case, the article management device 1200 may also obtain the ID recognized from the captured image including the display area 201 based on the template or the display area based on the tag 202, and the ID recognized from the captured image including the area where the information indicating the ID of the transport ship 400 is displayed, as the ID of the thick plate and the ID of the transport ship 400.

[0152] In addition, for example, the operator (operator) of the transport ship 400 may input the ID of the thick plate and the ID of the transport ship 400 into a handheld information processing terminal (tablet terminal, etc.). In this case, the ID of the thick plate and the ID of the transport ship 400 may be sent from the information processing terminal to the article management device 1200, so that the article management device 1200 obtains the ID of the thick plate and the ID of the transport ship 400.

[0153] Furthermore, when the thick plates are loaded onto the transport vehicle 700, the article management device 1200 may store information indicating the ID of the transport vehicle 700, for example, as follows.

[0154] First, the article management device 1200 obtains the ID of the plate and the ID (for example, license plate information) of the transport vehicle 700. Then, the article management device 1200 identifies a column storing information indicating an ID identical to the ID of the obtained plate from the ID column of the management table 1300. Then, the article management device 1200 stores information indicating the ID of the obtained transport vehicle 700 in the transport tool ID column of the identified column.

[0155] For example, the first imaging devices 430o to 430q may capture an image including the display area 201 based on the template or the display area based on the label 202 and an area displaying information indicating the ID of the transport vehicle 700. In this case, the article management device 1200 may also recognize and obtain the ID of the thick plate and the ID of the transport vehicle 700 from the captured image.

[0156] In addition, the first camera 430o to 430q may capture an image including the display area 201 based on the template or the display area based on the tag 202, and an image including the area where the information indicating the ID of the transport vehicle 700 is displayed. For example, after capturing the image including the display area 201 based on the template or the display area based on the tag 202, the image including the area where the information indicating the ID of the transport vehicle 700 is displayed may be captured. In this case, the article management device 1200 may also obtain the ID recognized from the captured image including the display area 201 based on the template or the display area based on the tag 202, and the ID recognized from the captured image including the area where the information indicating the ID of the transport vehicle 700 is displayed, as the ID of the thick plate and the ID of the transport vehicle 700.

[0157] In addition, for example, the operator (operator) of the transport vehicle 700 may input the ID of the thick plate and the ID of the transport vehicle 700 into a handheld information processing terminal (tablet terminal, etc.). In this case, the ID of the thick plate and the ID of the transport vehicle 700 may be sent from the information processing terminal to the article management device 1200, so that the article management device 1200 obtains the ID of the thick plate and the ID of the transport vehicle 700.

[0158] The data structure of the management table 1300 may be a data structure in which the information stored in each column of the management table 1300 (for example, the position of the item and the registration time) is updated every time the thick plate is placed at a new position. In addition, the data structure of the management table 1300 may be a data structure in which additional information is added. In the latter case, there is an advantage that the history of the position of the item can be added.

[0159] Next, an example of the contents stored in the on-hand stock display column of the management table 1300 and the contents stored in the item position column of the non-on-hand stock display column will be described in more detail.

[0160] Fig.14 FIG. 1 is a diagram showing an example of the position of a template within an article and the position of the components of each display item in a display based on the template. Fig.14 In the example, Figure 2 The display position shown and Figure 3A The content shown is displayed based on a template.

[0161] exist Fig.14 In the example, the position of the template in the object and the position of the components of each display item based on the template are represented by xy coordinates, which represent the eight vertices of the thick plate of the rectangular parallelepiped. Figure 2The vertex 203 shown is used as the origin, and the direction from the vertex 203 along the width direction of the thick plate is defined as the positive direction of the y-axis, and the direction along the length direction of the thick plate is defined as the positive direction of the x-axis.

[0162] In this embodiment, the xy coordinates of two vertices located on the diagonal line among the four vertices of the circumscribed rectangle determined by the positions of the components of each display item in the display based on the template, which are the maximum value of the x-axis, the minimum value of the x-axis, the maximum value of the y-axis, and the minimum value of the y-axis, are used as the position within the template. Fig.14 In FIG. 1 , the xy coordinates (Ls10, Ws10) and (Ls11, Ws11) of the two vertices 1401a and 1401b of the circumscribed rectangle are shown as the positions within the object of the template. In addition, for the sake of simplicity, the circumscribed rectangle is the coordinate axis representing the position within the object (in Fig.14 The example shown is a rectangle in which the x-axis and the y-axis are parallel.

[0163] In addition, the xy coordinates of the center of gravity (or center of image) of the component elements (characters, numbers, symbols, marks) of each display item in the display based on the template are used as the position of the component element. Fig.14 As shown in the figure, the y-axis coordinates of the components of ID305 included in the display based on the template, namely 5, 6, 3, 4, 9, 7, -, 0, and 1, are Ws1, Ws2, Ws3, Ws4, Ws5, Ws6, Ws7, Ws8, and Ws9, respectively, and the x-axis coordinates are all Ls1.

[0164] As described above, the position of each component of each display item displayed based on the template is expressed by the xy coordinates.

[0165] Fig.15 FIG. 1 is a diagram showing an example of the position of a tag within an article and the position of components of each display item in a display based on the tag. Fig.15 In the example, Figure 2 The display position shown and Figure 3B The display contents shown are displayed based on tags.

[0166] exist Fig.15 In the example, the position of the label in the item and the position of the components of each display item based on the label are represented by the yz coordinate, which represents the position of the eight vertices of the rectangular thick plate. Figure 2 The vertex 204 shown is used as the origin, the direction from the vertex 204 along the width direction of the thick plate is defined as the positive direction of the y-axis, and the direction along the plate thickness direction of the thick plate is defined as the positive direction of the z-axis.

[0167] In this embodiment, the label is rectangular. The yz coordinates of the two diagonal vertices of the four vertices of the label are used as the position of the label in the item. In addition, the yz coordinates of the center of gravity (or centroid) position of the component element of each display item in the display based on the label are used as the position of the component element. Fig.15 , the yz coordinates of the two vertices 1501a and 1501b of the tag are the positions of the tag within the article.

[0168] As mentioned above, in Fig.13 In the management table 1300 shown in FIG. 1 , the ID column of the template column of the current stock display column stores information indicating the content and position of the component of ID 305 included in the display based on the template. Fig.16A As shown, in Fig.14 In the example shown, "5", "6", "3", "4", "9", "7", "-", "0", "1" are stored in the ID column of the template column of the spot display column as the content of the component of ID305 included in the display based on the template. In addition, (Ls1, Ws1), (Ls2, Ws1), (Ls3, Ws1), (Ls4, Ws1), (Ls5, Ws1), (Ls6, Ws1), (Ls7, Ws1), (Ls8, Ws1), (Ls9, Ws1) are stored in the ID column of the template column of the spot display column as the position of the component.

[0169] In the other display item column of the stock display column, the same as the ID column of the template column, information indicating the content and position of the components of the display item is stored. In addition, the information indicating the position of the components of each display item in the label column is not the xy coordinate, but the yz coordinate.

[0170] In addition, if Fig. 16B As shown, in Fig.14 In the example shown, (Ls10, Ws10) and (Ls11, Ws11) are stored in the template column of the in-item position column of the non-stock display column as information indicating the in-item position of the template.

[0171] The label column of the article position column also stores information indicating the article position of the label, similar to the template column of the article position column. In addition, as described above, the information indicating the article position of the label is not the xy coordinate, but the yz coordinate.

[0172] When creating the management table 1300, for example, an operator may operate the user interface of the article management device 1200 to input information to be stored in each column of the management table 1300 into the article management device 1200. In addition, the article management device 1200 may receive information to be stored in each column of the management table 1300 from an external device such as a computer that manages factory operations.

[0173] The information stored in the stock display column of the management table 1300 and the information stored in the item position and main ID column of the non-stock display column are sometimes determined after the display for the thick plate is performed, and sometimes determined before the display for the thick plate is performed. Therefore, when the information to be stored in the column is determined before the display for the thick plate is performed, the item management device 1200 obtains the information to be stored in the column as described above, and stores it in each column of the management table 1300.

[0174] On the other hand, the information stored in the item position and registration time column of the non-stock display column cannot be determined before the display of the thick plate is performed. Therefore, the item management device 1200 obtains the information to be stored in the column and stores it in each column of the management table 1300, for example, when the information to be stored in the column is obtained. As described above, the date and time when the information stored in the item position column is changed is stored in the registration time column. In this case, the date and time when the thick plate is placed at the current position is registered in the registration time column.

[0175] In addition, with respect to items in the non-stock display column, information is stored in the item column only when the information of the item is obtained, and information is not stored in the item column when the information of the item is not obtained. In addition, even if the information stored in the item position column has not been updated after a specified time has passed from the date and time stored in the registration time column, the item management device 1200 determines that the determination of the thick plate position is interrupted, and preferably resets the item position and the information stored in the registration time column. In addition, at the same position, a thick plate with an earlier date and time stored in the registration time column will not be conveyed before a thick plate with a later date and time stored in the registration time column (that is, when there is overlap of plates, the plate located below will not suddenly disappear). Therefore, when the item management device 1200 performs the reset of the item position and the information stored in the registration time column, it is preferably performed as follows. That is, when there is a thick plate in the registration time column that stores a date and time that is later than the date and time recorded in the registration time column to be reset, and the thick plate is at the same three-dimensional position as the three-dimensional position stored in the item position column to be reset, the item management device 1200 preferably sends a signal for confirmation through an alarm or the like.

[0176] Next, an example of the function of the article management device 1200 is described. Fig.17 This is a flowchart for explaining an example of the processing of the article management device 1200 according to this embodiment. Fig.17 An example of the function of the article management device 1200 is described. In addition, as described above, in this embodiment, tracking is performed by the article management device 1200. Therefore, in this embodiment, Fig.17 The article management method implemented by the illustrated flowchart is a tracking method.

[0177] exist Fig.17 In step S1701, the first image acquisition unit 1201 acquires the first captured image. The first captured image is a captured image captured by any one of the first capturing devices 430a~430y, the first capturing devices possessed by the unmanned aerial vehicles 440a~440c, and the first capturing devices held by the operator (worker). The first captured image can be sent directly from the first capturing device to the item management device 1200, or can be sent from the first capturing device to the item management device 1200 via the information terminal device. Whenever the item management device 1200 receives a captured image captured by any one of the first capturing devices 430a~430y, the first capturing devices possessed by the unmanned aerial vehicles 440a~440c, and the first capturing devices held by the operator (worker), it executes the control based on Fig.17 In the present embodiment, the ID of the first imaging device that captured the first captured image and the imaging conditions when the first captured image was captured are transmitted together with the first captured image.

[0178] When acquiring the first captured image, the first image acquisition unit 1201 acquires the ID of the first imaging device that captured the first captured image, the imaging conditions when the first captured image was captured, and the date and time when the first captured image was captured. The imaging conditions include information set by the imaging device when capturing the image, which is necessary for determining the position of the display area based on the template or the display area based on the label as described later.

[0179] In addition, in the case where the position of the first camera when the first captured image is captured is not fixed, the position information of the first camera when the first captured image is captured is sent together with the first captured image. For example, in the case where the unmanned aerial vehicles 440a~440c can measure the three-dimensional coordinates (xyz coordinates) of the unmanned aerial vehicles using GPS (Global Positioning System), the three-dimensional coordinates can also be sent as the position information of the first camera of the unmanned aerial vehicle. Regarding the first camera 430n installed on the transport ship 400, the first camera 430g~430m installed on the loading and unloading device 411, and the first camera 430q installed on the transport vehicle 700, the position information of the first camera can be sent together with the first captured image, similar to the unmanned aerial vehicles 440a~440c. This situation is also the same for the first camera held by the operator (operator).

[0180] In addition, when the first camera cannot capture the area that should be included in the first captured image, the shooting conditions of the first camera (shooting direction, field angle, zoom ratio, aperture (F value), etc.) are adjusted before the processing of step S1701.

[0181] Next, in step S1702, the recognition unit 1202 recognizes the constituent elements (characters, numbers, signs, and marks) of the display items in the display based on the template (label) from the first captured image obtained in step S1701. The process of recognizing the content of the constituent elements of the display items can be realized by using the well-known OCR (Optical Character Recognition) technology, so its detailed description is omitted here. In addition, the process of step S1702 can also be realized by using AI (Artificial Intelligence) technology such as NN (Neural Network) or a technology combining AI and OCR (so-called AIOCR).

[0182] In addition, the recognition unit 1202 determines whether the constituent element of the display item is a template-based display or a label-based display. For example, the recognition unit 1202 performs edge detection processing based on the first captured image obtained in step S1701, detects the edge of the thick plate, and determines the area of ​​the thick plate based on the detected result. Then, the recognition unit 1202 determines whether the determined area is a plate surface portion of the thick plate or a side surface (plate thickness) portion of the thick plate. This determination is achieved, for example, based on the aspect ratio of the determined area. As a result of the determination, in the case where the determined area is the plate surface portion of the thick plate, the recognition unit 1202 recognizes that the constituent element of the display item is a template-based display constituent element. On the other hand, in the case where the determined area is a side surface (plate thickness) portion of the thick plate, the recognition unit 1202 recognizes that the constituent element of the display item is a label-based display constituent element.

[0183] In addition, the recognition unit 1202 can also perform edge detection processing based on the first captured image. When a rectangular label 202 area is detected, it is recognized that the constituent elements of the display item are constituent elements of the label-based display. Otherwise, it is recognized that the constituent elements of the display item are constituent elements of the template-based display.

[0184] In the present embodiment, for simplicity of description, for the thick plate located at the uppermost stage, the first captured image does not include a display area based on the label 202 but includes a display area based on the template 201 .

[0185] In addition, the recognition unit 1202 derives the position of the display item. An example of a method for the first recognition unit 1202 to derive the position of the display item is described. First, the recognition unit 1202 performs edge detection processing based on the first captured image obtained in step S1701 to detect the edge of the thick plate. Then, the recognition unit 1202 detects the origin of the position within the object in the first captured image (the prescribed position in the thick plate). The recognition unit 1202 derives the vertex that becomes the origin of the position within the object in the captured image (the prescribed position in the thick plate) from, for example, the 8 vertices of the thick plate of a rectangular parallelepiped, based on a positional relationship that is pre-set as a positional relationship between a display area based on a template (label) and the origin of the position within the object.

[0186] As the positional relationship between the display area based on the template (label) and the origin of the position in the item, for example, the following information can be pre-set in the item management device 1200. That is, the following information can be pre-set in the item management device 1200: when the top of the component element of the display item based on the display of the template (label) is viewed in a manner such that the top of the component element is located on the upper side, the vertex located on the lower right side of the display based on the template (label) in the same plane as the display based on the template (label) is the origin of the position in the item. The recognition unit 1202 derives the position based on the origin of the position in the item in the captured image as the position of the component element of the display item in the display based on the template (label). When the processing of step S1702 is completed, the processing of step S1703 is executed.

[0187] In step S1703 , the acquisition unit 1203 acquires information stored in the ID column of the current stock display column in the management table 1300 stored in the storage unit 1207 .

[0188] Next, in step S1704, the comparison unit 1204 compares the information recognized in step S1702 with the information acquired in step S1703. When it is recognized that the information recognized in step S1702 is information displayed based on a template, the comparison unit 1204 compares the information stored in the ID column of the template column of the stock display column among the information acquired in step S1703 with the information recognized in step S1702. On the other hand, when it is recognized that the information recognized in step S1702 is information displayed based on a tag, the comparison unit 1204 compares the information stored in the ID column of the tag column of the stock display column among the information acquired in step S1703 with the information recognized in step S1702.

[0189] Next, in step S1705, the identification unit 1205 identifies one slab from the slabs managed in the management table 1300 based on the comparison result in step S1703. Specifically, the identification unit 1205 identifies information that matches the information identified in step S1702 from the information acquired in step S1703.

[0190] Here, the so-called consistency means that the information identified in step S1702 contains a part whose content and arrangement order are completely consistent with the components stored in the template column of the spot display column or the ID column of the label column. Figure 3A as well as Figure 3BIn the example shown, the contents of the constituent elements of ID305 and 312 displayed by the template or label are 5, 6, 3, 4, 9, 7, -, 0, 1, and the arrangement order is 5, 6, 3, 4, 9, 7, -, 0, 1 from the left. Therefore, the determination unit 1205 determines the information in which 5, 6, 3, 4, 9, 7, -, 0, 1 are arranged in this order from the left from the ID column managed in the management table 1300. In the information identified in step S1702, not only when some of these constituent elements are missing, but also when constituent elements other than these constituent elements are arranged, the information identified in step S1702 is not information whose contents and arrangement order are completely consistent with the constituent elements stored in the ID column of the template column or the label column of the spot display column.

[0191] Next, in step S1706, the position deriving unit 1206 determines whether the position of the first camera that photographed the first captured image obtained in step S1701 is known based on the information obtained in step S1701. In the present embodiment, the three-dimensional positions of the first cameras 430a to 430f installed on the beam 412 and the leg structure 413, the first cameras 430o to 430p installed on the structure set at the standby position of the transport vehicle 700, the first cameras 430r to 430w installed on the structure set in the storage yard in the factory, and the first cameras 430x to 430y installed on the structure set in the production line of the factory are known. For each of these first cameras, the ID of the first camera and the position information (three-dimensional coordinates (xyz coordinates)) of the first camera are pre-registered in the article management device 1200 in association with each other. The position export unit 1206 determines that the position of the first camera that captured the first image obtained in step S1701 is known when position information (three-dimensional coordinates) associated with the same ID as the first camera obtained in step S1701 is pre-registered in the item management device 1200; otherwise, it determines that the position of the first camera that captured the first image obtained in step S1701 is unknown.

[0192] As a result of the determination in step S1706, if the position of the first camera that captured the first captured image acquired in step S1701 is not known, the process of step S1707 is performed. In step S1707, the position deriving unit 1206 determines whether the position information of the first camera that captured the first captured image was acquired in step S1701. As a result of the determination, if the position information of the first camera that captured the first captured image was not acquired in step S1701, the process proceeds to step S1712 described later.

[0193] When the position of the first camera that captured the first captured image obtained in step S1701 is known as a result of the determination in step S1706, and when the position information of the first camera that captured the first captured image is obtained in step S1701 as a result of the determination in step S1707, the process of step S1708 is performed. In step S1708, the position deriving unit 1206 determines whether the position in the height direction (z-axis direction) of the display area based on the template (label) can be determined. In addition, a specific example of the case where the position in the height direction (z-axis direction) of the display area based on the template (label) can be determined will be described later. As a result of the determination in step S1708, when the position in the height direction (z-axis direction) of the display area based on the template (label) cannot be determined, the process proceeds to step S1712 described later.

[0194] On the other hand, as a result of the determination in step S1708, when the position in the height direction (z-axis direction) of the display area based on the template (label) can be determined, the process of step S1709 is executed. In step S1709, the position deriving unit 1206 derives the three-dimensional coordinates (xyz coordinates) of the display area based on the template (label) as the three-dimensional position of the thick plate determined in step S1705.

[0195] Next, in step S1710, the storage unit 1207 updates the information stored in the item position column of the non-stock display column of the management table 1300 as information for the thick plate determined in step S1705 to the information indicating the three-dimensional position of the thick plate derived in step S1709. Thus, in the present embodiment, the case where the information stored in the item position column of the non-stock display column of the management table 1300 is the information display position is exemplified. In addition, in the present embodiment, the information indicating the three-dimensional position of the thick plate derived in step S1709 is an example of the information display position newly derived by the position deriving unit. In addition, the information stored in the item position column of the non-stock display column of the management table 1300 before the processing of step S1710 is an example of the information display position already stored in the storage medium. The information stored in the item position column of the non-stock display column of the management table 1300 is the xyz coordinate of the display area based on the template (label). Therefore, the information display position is a three-dimensional position. As described above, the information display position of the same slab changes not only the x-coordinate and the y-coordinate but also the z-coordinate (the position in the height direction) as the slab moves. In this embodiment, the position of the existing slab whose position has changed can be tracked by the processing of step S1710.

[0196] Next, in step S1711, the output unit 1208 outputs information indicating the three-dimensional position of the thick plate determined in step S1705. For example, the output unit 1208 causes a computer display to display information indicating the three-dimensional position of the thick plate determined in step S1705. Then, based on Fig.17 The processing of the flowchart is completed.

[0197] Here, with Figure 2 An example of a method for deriving the three-dimensional position of the display area based on the template (label) in step S1709 will be described using the display area 201 based on the template shown in FIG. Figure 2 In the example shown, the template-based display area 201 is an area near the center of the thick plate in the width direction (y-axis direction) of the thick plate in the plate surface area, and is located at one end side (negative direction side of the x-axis) in the length direction of the thick plate. In this embodiment, the three-dimensional position of the template-based display area 201 is an example of an information display position.

[0198] Fig.18 This is a diagram for explaining an example of a method of deriving the three-dimensional position of the display area 201 based on the template.

[0199] exist Fig.18 In the figure, point P represents the three-dimensional position of the first camera (the xyz coordinates of point P are (x p ,y p ,z p )). The three-dimensional position of the first camera is, for example, the principal point of the first camera. Fig.18 In the example shown, the z-axis is a direction perpendicular to the surface of the thick plate (the surface on which the display area 201 based on the template is formed). In addition, the y-axis is a direction parallel to the direction in which the components of the same display item are arranged in the display area 201 based on the template. In addition, the x-axis is a direction parallel to the direction in which different display items are arranged in the display area 201 based on the template. In addition, the x-axis, the y-axis, and the z-axis are perpendicular to each other. In addition, here, in order to simplify the description, it is assumed that the width direction of the thick plate is parallel to the y-axis, and the length direction of the thick plate is parallel to the x-axis.

[0200] Point S represents the position of the vertex located on the negative side of the x-axis and the positive side of the y-axis among the four vertices of the display area 201 based on the template (the xyz coordinates of point S are (x s ,y s ,z s )). Point O is the foot of the perpendicular line to point P (relative to the xy plane) (the intersection of the perpendicular line to point P and the xy plane). Fig.18 In the example shown, the origin is set to point O. Point Q is the intersection of a straight line passing through point O and parallel to the x-axis and a straight line passing through point S and parallel to the y-axis.

[0201] When ∠OPQ is β (rad) and the length of line segment OP (the distance from point O to point P) is H, the length L of line segment OQ (the distance from point O to point Q) is expressed by the following formula (1).

[0202] L=Htanβ……(1)

[0203] When ∠OPS is α (rad), the length of line segment OS (the length from point O to point S) l is expressed by the following equation (2). Also, the length of line segment SQ (the distance from point S to point Q) w y It is represented by the following formula (3).

[0204] l=Htanα……(2)

[0205] w y =(l 2 -L 2 ) 0.5 ……(3)

[0206] Therefore, the xyz coordinates of point S (x s ,y s ,z s ) is expressed by the following formula (4).

[0207] S(x s ,y s ,z s )=(L,w y ,0)……(4)

[0208] In this embodiment, the following example is given: when the length H of the line segment OP, ∠OPQ (=β), and ∠OPS (=α) are determined, the position derivation unit 1206 derives the xyz coordinates (x s ,y s ,z s ) as the position of the display area 201 based on the template.

[0209] The length H of the line segment OP is the z coordinate z of the point P. p Subtract the z coordinate z of the display area 201 based on the template s Therefore, once the z coordinate of the display area 201 based on the template (the position in the height direction (z-axis direction) of the display area 201 based on the template) is determined, the length H of the line segment OP can be derived. Fig.18 The z coordinate z of the display area 201 based on the template is illustrated in s = 0 (zero), so the length H of line segment OP is equal to the z coordinate z of point P. p equal.

[0210] The position of the point S in the first captured image is derived, for example, as follows. The position deriving unit 1206 derives a circumscribed rectangle determined by the positions of the maximum value of the x-axis, the minimum value of the x-axis, the maximum value of the y-axis, and the minimum value of the y-axis in the constituent elements of the template-based display item identified in step S1702. Then, the position deriving unit 1206 derives the positions of the vertices of the derived circumscribed rectangle, for example, the positions of the vertices located on the lower left side (negative side of the x-axis and positive side of the y-axis) when the top of the constituent elements of the template-based display item is located on the upper side, as the position of the point S in the first captured image. In addition, when label-based display is performed in the first captured image, the position deriving unit 1206 derives the vertices of the rectangular label, for example, the vertices located on the lower left side when the top of the constituent elements of the template-based display item is located on the upper side, as the position of the point S in the first captured image.

[0211] The position of point Q in the first captured image is a position on a straight line passing through the position of point S in the first captured image and parallel to the y-axis, and is a position corresponding to the center of the horizontal field of view (the field of view in the y-axis direction) at the time of shooting. In addition, the determination of point Q can be performed, for example, as follows. That is, by irradiating a laser from point P with a laser pointer or the like, a straight line passing through the position of point S and parallel to the y-axis is displayed on a surface of the plate surface of the thick plate on which a display area 201 based on a template is formed. In addition, by irradiating a laser from point P with a laser pointer or the like, a straight line passing through point O and parallel to the x-axis is displayed on a surface of the plate surface of the thick plate on which a display area 201 based on a template is formed. In this case, the intersection of the two straight lines displayed by the laser is point Q. Therefore, the position deriving unit 1206 can also determine point Q based on the first captured image captured in a state where the intersection of the two straight lines displayed by the laser is displayed. In this way, when points P, O and Q are located on the xz plane (on the plane of y=0), the position of point Q can be determined even if the surface of the thick plate where the template-based display area 201 is formed is inclined relative to the xy plane.

[0212] For example, the position deriving unit 1206 may derive ∠OPS (= α), ∠OPQ (= β), and ∠SPQ (= δ) as follows. First, the position deriving unit 1206 derives the lengths of the line segment PS and the point PQ based on the measurement value of the distance measuring sensor (e.g., a distance measuring sensor of the TOF (Time-of-Flight) method) possessed by the first imaging device. Then, the position deriving unit 1206 determines the positions of the point S and the point Q, assuming that the surface on which the display area 201 based on the template is formed is an xy plane and the line segment QS is a straight line along the y-axis direction. Then, the position deriving unit 1206 derives ∠OPS (= α), ∠OPQ (= β), and ∠SPQ (= δ) based on the positions of the point P, the point S, and the point Q.

[0213] In addition, for example, the position derivation unit 1206 can also derive ∠OPS (= α), ∠OPQ (= β), and ∠SPQ = (= δ) based on the angle of the laser irradiated from the first camera toward point S, the angle of the laser irradiated from the first camera toward point Q, and the scanning angle when the laser is scanned from point S to point Q by the first camera.

[0214] In addition, the position deriving unit 1206 can also derive ∠OPS (= α), ∠OPQ (= β), and ∠SPQ (= δ) by counting the number of rectangles arranged in a grid in the first captured image captured in a state where the surface of the plate surface where the display area 201 based on the template and the area of ​​the point O are formed is irradiated with a grid-shaped laser. In this case, the length of a rectangle in the x-axis direction is, for example, the length in the x-axis direction that changes when ∠OPS (= α) and ∠OPQ (= β) change by 1 rad, and the length in the y-axis direction is, for example, the length in the y-axis direction that changes when ∠SPQ (= δ) changes by 1 rad. Based on the count result of the number of rectangles from the rectangle located at the position corresponding to point O to the rectangles corresponding to points S and Q, ∠OPS (= α) and ∠OPQ (= β) are derived. In addition, based on the count result of the number of rectangles from the rectangle located at the position corresponding to point S to the rectangle located at the position corresponding to point Q, ∠SPQ (= δ) is derived. The number of rectangles is counted based on the result of the image processing on the first captured image.

[0215] In addition, the positions of the display areas 1111, 1121, 1131, and 1141 are also the same as those in the reference Fig.18 In addition, the display area 201 based on the template and the display area based on the label 202 are different only in the surface arranged on the thick plate, so the position of the display area based on the label 202 can also be imitated by referring to Fig.18 Therefore, the detailed description is omitted here.

[0216] Next, an example of a case where it is determined in step S1708 that the position of the display area in the height direction (z-axis direction) based on the template (label) can be determined will be described.

[0217] As a first example, first, the position deriving unit 1206 derives the distance in the height direction (z-axis direction) from the first imaging device to the display area based on the template using a laser range finder. Then, the position deriving unit 1206 derives the position obtained by subtracting the distance from the position in the height direction (z-axis direction) of the first imaging device as the position in the height direction (z-axis direction) of the display area 201 based on the template.

[0218] As a second example, the position deriving unit 1206 uses the information indicating the position of the thick plate stored in the item position column of the management table 1300 and the information indicating the date and time when the thick plate is placed at the current position stored in the registration time column of the management table 1300 to derive the position of the display area based on the template (label) in the height direction (z-axis direction). An example of a method for deriving the position of the display area based on the template (label) in the height direction (z-axis direction) is described. In addition, here, as needed, the three-dimensional position of the thick plate stored in the item position column of the management table 1300 is referred to as the item position. In addition, here, in order to simplify the description, it is assumed that the thick plate located at the bottom layer of the stacked multiple thick plates is placed on the ground.

[0219] First, based on the object positions stored in the management table 1300, the position export unit 1206 determines, from the thick plates other than the thick plates determined in step S1705, thick plates that are at least partially included in the shooting range (xy coordinates) of the first shooting device that shot the first shooting image obtained in step S1701.

[0220] When the position in the height direction (z-axis direction) of the display area 201 based on the template is derived, the thick plate identified in step S1705 is located at the uppermost layer. Therefore, the position deriving unit 1206 determines that the thick plate at least partly included in the imaging range (xy coordinates) of the first imaging device that captured the first imaging image acquired in step S1701 is located below the thick plate identified in step S1705. In this case, all the dimensions (thicknesses) stored in the management table 1300 as information on the thick plate located below the thick plate identified in step S1705 and the value obtained by adding the thickness of the thick plate identified in step S1705 become the position in the height direction (z-axis direction) of the display area 201 based on the template.

[0221] On the other hand, when deriving the position in the height direction (z-axis direction) based on the display area of ​​the tag, the position deriving unit 1206 determines that among the thick plates determined as described above, the thick plates for which the registration time column stores a date and time later than the date and time stored in the registration time column of the management table 1300 as the information of the thick plates determined in step S1705 are the thick plates located below the thick plates determined in step S1705. In this case, the value obtained by adding the sum of all the dimensions (thicknesses) stored in the management table 1300 as the information of the thick plates located below the thick plates determined in step S1705 and the value of the z coordinate of the position in the article stored in the management table 1300 as the information of the thick plates determined in step S1705 becomes the position in the height direction (z-axis direction) based on the display area of ​​the tag 202.

[0222] As a third example, when information capable of specifying the height direction (z-axis direction) is displayed in the first captured image, the position deriving unit 1206 can derive the position of the display area based on the template (label) in the height direction (z-axis direction) based on the information. An example of a method for deriving the position of the display area based on the template (label) in the height direction (z-axis direction) is described. For example, when the first captured image includes a structure having a scale indicating the position in the height direction (z-axis direction), the position deriving unit 1206 recognizes the scale of the structure from the first captured image. For example, the position deriving unit 1206 performs edge detection processing on the first captured image to extract the edge of the object, and extracts the structure from the extracted object. For example, information that associates information indicating the outline of the object with the name of the object is pre-registered in the article management device 1200. In this case, the position deriving unit 1206 can also extract the object corresponding to the structure by comparing the edge of the object extracted as described above with the information. Furthermore, the position derivation unit 1206 may extract the structure from the objects extracted as described above using AI (Artificial Intelligence) such as NN (Neural Network).

[0223] Then, the position deriving unit 1206 derives the position of the display area based on the template (label) in the height direction (z-axis direction) based on the position of the display area based on the template (label) in the first captured image and the position of the scale displayed on the structure in the first captured image. In addition, the numbers added to the scale in the first captured image are recognized by the same method as the method of recognizing the constituent elements of the display item based on the template (label).

[0224] As a fourth example, the position deriving unit 1206 may derive the position of the display area 201 in the height direction (z-axis direction) based on the template using the second captured image acquired by the second image acquiring unit 1209. Fig.17 In the flowchart of, before the timing of starting the processing of step S1708, a step of acquiring a second captured image by the second image acquisition unit 1209 is added. The second captured image is a captured image captured by a second imaging device different from the first imaging device (the imaging device that captures the area including the display area based on the template (label)) that captured the first captured image. The second captured image includes both the first imaging device that captured the first captured image acquired in step S1701 and the thick plate determined in step S1705. An example of a method of deriving the position of the display area 201 based on the template in the height direction (z-axis direction) in this way is described.

[0225] Fig.19 This is a diagram for explaining an example of a method of deriving the position of the display area 201 in the height direction (z-axis direction) based on the template.

[0226] exist Fig.19 In the example shown, Fig.18 Similarly, point P represents the position of the first camera (the xyz coordinates of point P are (x p ,y p ,z p )). In addition, the z-axis is a direction perpendicular to the surface of the thick plate (the surface on which the display area 201 based on the template is formed). In addition, the y-axis is a direction parallel to the direction in which the components of the same display item in the display area 201 based on the template are arranged. In addition, the x-axis is a direction parallel to the direction in which different display items in the display area 201 based on the template are arranged. In addition, the x-axis, the y-axis, and the z-axis are perpendicular to each other. Here, in order to simplify the explanation, it is also assumed that the width direction of the thick plate is parallel to the y-axis, and the length direction of the thick plate is parallel to the x-axis.

[0227] Point O' is the intersection of a perpendicular line to point P (with respect to the xy plane) and a plane including the surface of the thick plate (the surface where the display area 201 based on the template is formed) (point O' and Fig.18 Point O is the foot of the perpendicular line to point P (relative to the xy plane) (the intersection of the perpendicular line to point P and the xy plane). Fig.19 In the example shown, point O is the origin. Point R represents the three-dimensional position of the second imaging device (the xyz coordinates of point R are (x r ,y r ,z r )). The three-dimensional position of the second imaging device is, for example, the position of the principal point of the second imaging device.

[0228] The three-dimensional position of the second camera can be determined by the article management device 1200. For example, when the three-dimensional position of the second camera is fixed, the ID of the second camera and the three-dimensional coordinates (xyz coordinates) of the second camera are associated with each other and pre-registered in the article management device 1200. In addition, when the three-dimensional position of the second camera is not fixed, for example, the second camera sends the three-dimensional coordinates (xyz coordinates) of the second camera measured using GPS and the ID of the second camera. In this case, the second image acquisition unit 1209 acquires this information sent from the second camera.

[0229] exist Fig.19 In the example shown, for simplicity of description, it is assumed that the bottommost thick plate among the stacked thick plates is placed on the ground. Point D is the vertex closest to point R among the vertices of the surface forming the display area 201 based on the template. Point B is the vertex closest to point R among the vertices of the bottom surface of the bottommost thick plate among the stacked thick plates.

[0230] exist Fig.19 middle, as well as For example, it is determined based on the positions of point P, point B, and point D in the second captured image and the principal point of the second captured image. The direction in which the line segment RP extends is determined by a straight line connecting point P in the second captured image and the principal point of the second captured image. The direction in which the line segment RB extends is determined by a straight line connecting point B in the second captured image and the principal point of the second captured image. The direction in which the line segment RT extends is determined by a straight line connecting point D in the second captured image and the principal point of the second captured image. It is determined according to the direction in which line segment RB extends and the direction in which line segment RD extends. It is determined according to the direction in which the line segment RP extends and the y-axis direction.

[0231] Furthermore, for example, the position deriving unit 1206 derives the length of the line segment RP (the distance from point R to point P) and the length of the line segment RD (the distance from point R to point D) based on the measurement value of the distance measuring sensor (e.g., a distance measuring sensor of the TOF (Time-of-Flight) method) included in the second camera. Then, the position deriving unit 1206 derives the length of the line segment RP (the distance from point R to point P) and the length of the line segment RD (the distance from point R to point D) based on the position (zr) of the second camera in the height direction (z-axis direction), the length of the line segment RP and The position (z axis direction) of the first imaging device is derived. p )(z p =z r + Line segment RP Similarly, the position deriving unit 1206 determines the position of the second imaging device in the height direction (z-axis direction) based on the position (z r ), the length of line segment RD, and The position of the display area 201 in the height direction (z-axis direction) based on the template is derived.

[0232] In addition, Fig.19 , the case where the position of the second camera (point R) in the height direction (z-axis direction) is 0 (zero) is illustrated. In addition, as described above, if the second captured image is used, the position deriving unit 1206 can derive not only the position of the display area 201 in the height direction (z-axis direction) based on the template, but also the position of the first camera in the height direction (z-axis direction) (z p ). In addition, the position deriving unit 1206 can derive not only the position in the height direction (z-axis direction) but also the position in the x-axis direction (x-axis direction) of the first imaging device. p ) and the position in the y-axis direction (y p ). For example, according to the direction in which the line segment RP extends and the y r + Line segment RP To export x p ,y p That is, according to the direction in which the line segment RP extends and the y r + Line segment RP Derive the vector connecting points R and P. The length of the x-axis component and the length of the y-axis component of this vector are x p ,y p .

[0233] In addition, here, the case where the second captured image captured by the second camera includes point P (first camera) is described as an example. However, as long as the relative positional relationship between the first camera (point P) and the second camera (point R) can be derived ( etc.), it is not necessarily necessary to do so. In this case, for example, a captured image captured by at least one camera different from the first camera and the second camera is used. A specific example in this case is described below. First, a third camera is provided in addition to the first camera and the second camera. The third camera includes point P (first camera) and point R (second camera) in the shooting range, and excludes point D from the shooting range. In addition, the point representing the position of the third camera is set to R', which is not shown. In addition, one or both of the position of point R representing the position of the second camera and the position of point R' representing the position of the third camera are predetermined. In this case, the position derivation unit 1206 derives the elevation angle or depression angle when observing point R and point P from point R' based on the third captured image captured by the third camera, and then the position derivation unit 1206 derives the elevation angle or depression angle based on the derived elevation angle or depression angle. Angle, direction and length are derived by reference Fig.19 The method described above is implemented in the same manner as described above. In addition, when the shooting range of the third camera includes point P (first camera) but does not include point R (second camera), for example, it is sufficient to further use a fourth captured image captured by a fourth camera that includes point R' (third camera) and point R (second camera) in the shooting range. On the other hand, for example, when the shooting range of the third camera includes point R (second camera) but does not include point P (first camera), for example, it is sufficient to further use a fifth captured image captured by a fifth camera that includes point R' (third camera) and point P (first camera) in the shooting range.

[0234] In addition, the position derivation unit 1206 may determine the position of point P (the first camera) by using the sixth captured image captured by the sixth camera that includes the first camera in the capturing range. In this case, it is assumed that the sixth camera is included in the capturing range of at least one of the second camera, the third camera, and the fourth camera. In addition, the point representing the position of the sixth camera is set to P', which is not shown. In addition, one or both of the position of point P representing the position of the first camera and the position of point P' representing the position of the sixth camera are predetermined. In this case, for example, in the above description, the sixth camera and point P' are used instead of the first camera and point P for calculation. Then, for example, based on the result of the calculation, the sixth captured image, and the position of one or both of point P and point P', the relative position relationship ( wait).

[0235] As described above, even when the second imaging device cannot be configured to include both the first imaging device and the thick plate within the imaging range of the second imaging device, the position of the display area 201 in the height direction (z-axis direction) based on the template can be derived. In addition, the third to sixth captured images are acquired by, for example, the second image acquisition unit 1209. However, in addition to the second image acquisition unit 1209, the third to sixth image acquisition units that separately acquire the third to sixth captured images may also be used. In this case, Fig.17 The following step is added to the flowchart of: separately acquiring the third to sixth captured images at a timing earlier than the timing of starting the process of step S1708 and at a timing different from the timing of executing the step of acquiring the second captured image by the second image acquisition unit 1209. In addition, in the case where the position of the display area in the height direction (z-axis direction) based on the template (label) can be determined even without acquiring the second captured image, the second image acquisition unit 1209 is not required.

[0236] As a fifth example, the position deriving unit 1206 may derive the position of the thick plates 1002a and 1002b placed on the conveying table 1001 in the height direction (z-axis direction) of the display area 201 of the template, based on the position of the conveying table 1001 and the size (thickness) stored in the management table 1300 as information of the thick plates 1000a and 1002b. In this case, the position of the conveying table 1001 is registered in advance in the article management device 1200.

[0237] For example, in at least one of the above examples, when the position of the height direction (z-axis direction) of the display area based on the template (label) can be derived, it is determined in step S1708 that the position of the height direction (z-axis direction) of the display area based on the template (label) can be determined.

[0238] Furthermore, when the thick plate is located on the transport tool, in step S1709, the position deriving unit 1206 obtains the three-dimensional position of the thick plate on the transport tool. In the present embodiment, the transport tool is the transport ship 400 and the transport vehicle 700. Therefore, in the present embodiment, the case where the transport tool is the transport ship 400 and the transport vehicle 700 is illustrated.

[0239] For example, when a thick plate is loaded onto the transport ship 400, the position deriving unit 1206 obtains the ID of the thick plate and the information indicating the three-dimensional position of the thick plate in the transport ship 400. Then, the position deriving unit 1206 determines the column storing the information indicating the same ID as the ID of the obtained thick plate from the ID column of the management table 1300. Then, the position deriving unit 1206 stores the obtained information indicating the three-dimensional position of the thick plate in the transport ship 400 in the article position column of the determined column. For example, the operator (operator) of the transport ship 400 may also input the ID of the thick plate and the information indicating the three-dimensional position of the thick plate in the transport ship 400 into a handheld information processing terminal (tablet terminal, etc.). In this case, by sending the ID of the thick plate and the information indicating the three-dimensional position of the thick plate in the transport ship 400 from the information processing terminal to the article management device 1200, the position deriving unit 1206 may also obtain the ID of the thick plate and the information indicating the three-dimensional position of the thick plate in the transport ship 400. The information indicating the three-dimensional position of the thick plate is, for example, the xyz coordinates of the display area based on the template (label).

[0240] Furthermore, when a thick plate is loaded onto the transport vehicle 700, the position deriving unit 1206 obtains the ID of the thick plate and the information indicating the three-dimensional position of the thick plate in the transport vehicle 700. Then, the position deriving unit 1206 determines the column storing the information indicating the same ID as the ID of the obtained thick plate from the ID column of the management table 1300. Then, the position deriving unit 1206 stores the obtained information indicating the three-dimensional position of the thick plate in the transport vehicle 700 in the item position column of the determined column. For example, the operator (operator) of the transport vehicle 700 may also input the ID of the thick plate and the information indicating the three-dimensional position of the thick plate in the transport vehicle 700 into a handheld information processing terminal (tablet terminal, etc.). In this case, by sending the ID of the thick plate and the information indicating the three-dimensional position of the thick plate in the transport vehicle 700 from the information processing terminal to the item management device 1200, the position deriving unit 1206 may also obtain the ID of the thick plate and the information indicating the three-dimensional position of the thick plate in the transport vehicle 700. The information indicating the three-dimensional position of the thick plate is, for example, the xyz coordinates of the display area based on the template (label).

[0241] As described above, by acquiring the three-dimensional position of the thick plate in the conveying tool, the conveying tool that conveys each thick plate can be confirmed. In addition, the three-dimensional position of the thick plate when being conveyed by the conveying tool can be accurately determined.

[0242] As a sixth example, a photographing device in which the number of pixels and size of the photographing element (the length of each pixel [m / pixel]) and the distance from the photographing element to the principal point, i.e., the focal distance, are known can be used to obtain the position of the display area based on the template (label) in the actual space, thereby deriving the position of the display area based on the template (label) in the height direction (z-axis direction).

[0243] For example, the template-based display area 201 is a rectangle, and the actual lengths of the long side and the short side are pre-stored in the item management device 1200. Fig. 20A and Fig. 20B The sixth example is described below. As described above, information on the number of pixels of the sensor of the first imaging device, information on the size of the sensor, and focal lengths for each optical magnification used during imaging are stored in advance in the item management device 1200. The focal length is information determined during imaging.

[0244] Fig. 20A FIG. 2 is a diagram showing an example of a first captured image including a display area 201 based on a template. The display area 201 based on the template is a rectangular shape in the first captured image when the image is captured from the front. On the other hand, the display area 201 based on the template is sometimes a rectangular shape when the image is captured from an oblique direction. Fig. 20A In the first captured image, it becomes an irregular trapezoid as shown by the dotted line (imaginary line). Fig. 20B 2 is a diagram showing an example of the geometric relationship between the interior of the first imaging device and the display area 201 based on the template. Fig. 20B In the present invention, first, the position deriving unit 1206 determines the positions of the vertices 2002a to 2002d of the four corners of the display area 201' based on the template captured by the image sensor 2002. Next, the position deriving unit 1206 determines the straight lines 2003a to 2003d obtained by extending the straight line connecting the vertices 2002a to 2002d of the four corners of the display area 201' based on the template and the principal point FP to the display area 201 based on the template. Next, the position deriving unit 1206 determines the following square cone 2001: the principal point FP is the head vertex, the straight lines 2003a to 2003d are four sides with the principal point FP as one end, and the size of the bottom surface is the actual space length of the long side and the short side of the display area 201 (rectangle) based on the template. As described above, when the display area 201 based on the template is set as a rectangle, the actual space length of the long side and the short side of the rectangle is stored in advance in the item management device 1200. The position derivation unit 1206 derives the three-dimensional positional relationship between the display area 201 based on the template and the principal point FP (first imaging device) from the pyramid 2001 thus determined.

[0245] Here, for ease of explanation, the case where the shape of the display area based on the template is a rectangle of a known size in advance is exemplified. However, in the method of the sixth example, the shape of the display area that is the object of deriving the three-dimensional positional relationship with the first camera is not limited to the case of a rectangle. For example, even if the display is a text, a sign, a mark, etc. of a known shape and size in advance, as long as the shape and size of the display area are Fig. 20B By replacing the display area 201 based on the template shown with the display area of ​​this display, it is possible to derive the three-dimensional positional relationship between the position of this display and the principal point (first imaging device).

[0246] Furthermore, here, an example is given of a case where the display area that is the object of deriving the three-dimensional positional relationship with the first camera is a plane. However, in the method of the sixth example, the display area that is the object of deriving the three-dimensional positional relationship with the first camera is not limited to a plane. As long as the shape of the surface of the display area is known, the three-dimensional positional relationship between the display area and the first camera can be derived. For example, in a case where the position deriving unit 1206 derives the three-dimensional positional relationship between the display area displayed on the surface of a cylindrical object and the first camera, it is sufficient to replace the reference Fig. 20B The shape and size of the display area and the curvature of the display surface of the display area can be stored in the article management device 1200 in advance. The reason is that the information can be determined with reference to Fig. 20B A surface corresponding to the bottom surface of the square pyramid in the method of the sixth example described.

[0247] return Fig.17 As described above, if it is determined in step S1708 that the position in the height direction (z-axis direction) of the display area based on the template (label) cannot be determined, the processing of step S1712 is performed. In step S1712, the output unit 1208 outputs information indicating that the three-dimensional position of the thick plate cannot be determined and the ID of the first camera that photographed the thick plate. For example, the output unit 1208 causes the computer display to display information indicating that the three-dimensional position of the thick plate cannot be determined and the ID of the first camera that photographed the thick plate. When this information is output, the operator (operator) goes to the location where the corresponding first camera exists to determine the three-dimensional position of the corresponding thick plate. In addition, the operator (operator) can also determine the three-dimensional position of the corresponding thick plate based on the preview image of the corresponding first camera.

[0248] Next, in step S1713, the storage unit 1207 inputs information indicating the three-dimensional position of the thick plate determined by the operator (worker), and updates the information stored in the item position column of the non-stock display column of the management table 1300 as information for the thick plate to the input information. In addition, as a method for inputting information indicating the three-dimensional position of the thick plate, at least one of input based on the operation of the user interface of the item management device 1200, reception of information sent from an external device, and reading of information stored in a removable storage medium can be adopted. Then, Fig.17 The processing of the flowchart is completed.

[0249] <Conclusion>

[0250] As described above, in the present embodiment, the article management device 1200 obtains the first captured image of the area including the display area, which displays the information displayed on the thick plate through the template (label), and is information managed by the manufacturer of the thick plate. The article management device 1200 derives the three-dimensional position of the display area based on the template (label) according to the first captured image obtained and the position of the first shooting device that shot the first captured image, and stores the information representing the three-dimensional position in the storage medium in association with the ID of the thick plate. Thus, for example, even in the case where the thick plate is transported using various conveying devices (transport ships 400, transport vehicles 700, conveying workbenches 1001, etc.) or transported by operators, in the case where the thick plate is placed in a storage field where the address of the storage field is not determined, and in the case where the thick plate is stacked, the position of the thick plate can be managed. Therefore, the position of the thick plate can be properly managed.

[0251] In addition, in this embodiment, when the three-dimensional position of the display area based on the template (label) and the ID of the thick plate are already stored in the storage medium in association with each other, if the three-dimensional position of the display area based on the template (label) is newly derived, the three-dimensional position of the display area based on the template (label) already stored in the storage medium is updated to the newly derived three-dimensional position. Thus, it is possible to improve the tracking accuracy of the position of the thick plate whose position (location) changes with the passage of time.

[0252] Furthermore, in the present embodiment, the article management device 1200 recognizes the information (ID 305, 312) displayed in the display area based on the template (label) based on the first captured image. Then, the article management device 1200 compares the recognized information with the information (ID) stored in the management table 1300, and determines the thick plate based on the comparison result. Therefore, the thick plate itself that is the object of position management can be reliably determined based on the first captured image. Therefore, for example, compared with the case where the three-dimensional position of the thick plate is determined based on the action of the conveying device, the position of the thick plate can be determined with high accuracy.

[0253] Furthermore, in the present embodiment, the article management device 1200 obtains a second captured image captured by a second camera device different from the first camera device that captured the first captured image. The second captured image includes the first captured image and a thick plate (which is the object of position management). The article management device 1200 derives the position of the display area 201 based on the template in the height direction (z-axis direction) based on the obtained second captured image and the position of the second camera device. Then, the article management device 1200 derives the three-dimensional position of the display area 201 based on the template as the three-dimensional position of the thick plate based on the position of the display area 201 based on the template in the height direction (z-axis direction), the first captured image, and the position of the first camera device that captured the first captured image. Therefore, even in the case where the position of the thick plate in the height direction cannot be derived using the first captured image or the sensor possessed by the first captured image, the position of the thick plate can be managed.

[0254] In addition, in this embodiment, the display based on the template and the display based on the label are set as information that can be recognized by humans. Therefore, it becomes easy for operators (operators) to manage the inventory. Therefore, the convenience of inventory management is improved.

[0255] In addition, in this embodiment, information determined individually for each thick plate (information unique to the thick plate) is included in the display based on the template and the display based on the label. Therefore, the thick plate can be uniquely identified based on the first captured image. Therefore, the thick plate can be easily and quickly identified.

[0256] In this embodiment, the article management device 1200 derives three-dimensional coordinates (xyz coordinates) as the position of the display area based on the template (label). Therefore, for example, even when a plurality of thick plates are stacked, the positions of the thick plates can be managed.

[0257] <Modification>

[0258] <<First Modification>>

[0259] The method of deriving the position of the display area based on the template (label) included in the first captured image in step S1709 is not limited to referring to Fig.18 For example, the position deriving unit 1206 derives the position of the origin when determining the position within the item in the management table 1300 from the first captured image. Then, the position deriving unit 1206 derives the position of the display area based on the template (label) included in the first captured image based on the derived position of the origin and the information stored in the column of the position within the item in the management table 1300. An example of a method for deriving the position of the display area based on the template (label) included in the first captured image in this way is described by taking the display area 201 based on the template as an example.

[0260] Fig.21 This is a diagram for explaining an example of a method for deriving the position of the display area 201 based on a template. Fig.18 same, Fig.21 The xyz coordinates S(x s ,y s ,z s ) as the position of the display area 201 based on the template. Fig.14 As explained above, the position of the template in the object is represented by xy coordinates, which are Figure 2 The vertex 203 shown is the origin, the direction from the vertex 203 along the width direction of the thick plate is set as the positive direction of the y-axis, and the direction along the length direction of the thick plate is set as the positive direction of the x-axis. Fig.21 The position of point E becomes the origin of the position in the template. Point F is a point among the vertices of the thick plate that is arranged in the width direction (y-axis direction) relative to point E (points E and F form a side along the width direction of the thick plate). Point T is the foot of the perpendicular line of point P to line segment EF (the intersection of the perpendicular line of point P and line segment EF). Point P and point O are Fig.18 Same as shown.

[0261] exist Fig.21 In the example, the length of line segment PT is set to h p . In addition, the length of line segment PF is set to h1, ∠FPT is set to θ1 (rad), the length of line segment PE is set to h2, ∠EPT is set to θ2 (rad), and ∠EPF is set to θ (rad) (θ = θ1 + θ2). In addition, the length of line segment FT is set to w 01 , set the length of line segment ET to w 02 .

[0262] So, the x-coordinate of point E is x e becomes the x-coordinate x of point P p Add the length l of line segment OT pThe obtained value (x e =x p +l p ). The y coordinate of point E e becomes the y coordinate y from point P p Subtract the length w of line segment ET o2 The obtained value (y e =y p -w o2 ). Therefore, we only need to derive the length l of line segment OT p and the length w of line segment ET o2 , the xy coordinates (x e ,y e ).

[0263] The length lp of the line segment OT is expressed by the following formula (5).

[0264] l p =√(h p 2 -H 2 )=√(h1 2 cos 2 θ1-H 2 )=√(h2 2 cos 2 θ2-H 2 )……(5)

[0265] The length w of line segment ET o2 It is represented by the following formula (6).

[0266] w o2 =h2sinθ2……(6)

[0267] Therefore, for example, if the length h2 of the line segment PE and ∠EPT (=θ2) are derived, the length l of the line segment OT can be derived. p and the length w of line segment ET o2 In addition, the length H of the line segment OP is as shown in Fig.18 Export as described.

[0268] The position of point T in the first captured image is a position on line segment EF in the first captured image, and is a position corresponding to the center of the horizontal field angle (field angle in the y-axis direction) during shooting. For example, ∠EPT (=θ2) is determined based on the positions of point E and point T in the first captured image obtained in step S1701, and the principal point of the first captured image. The direction in which line segment PE extends is determined by a straight line connecting point E in the first captured image and the principal point of the first captured image. The direction in which line segment PT extends is determined by a straight line connecting point T in the first captured image and the principal point of the first captured image. ∠EPT (=θ2) is determined based on the direction in which line segment PE extends and the direction in which line segment PT extends.

[0269] For example, the length h2 of the line segment PE is derived using a distance measuring sensor (eg, a TOF (Time-of-Flight) distance measuring sensor) included in the first imaging device. Alternatively, the length h2 of the line segment PE may be derived from the length H of the line segment OP and ∠OPE.

[0270] As above, the x coordinate x of point E is derived. e and the y coordinate y e Then, the position deriving unit 1206 reads the xy coordinates (Ls10, Ws10) of the position corresponding to point S from the information stored in the item position column of the management table 1300 as the information of the label of the thick plate determined in step S1705. Then, the position deriving unit 1206 derives the x coordinate x of point E. e The value obtained by adding LS10 is taken as the x coordinate x of point S s (x s =x e +LS10). In addition, the position deriving unit 1206 derives the y coordinate x of point E. y The value obtained by adding WS10 is taken as the y coordinate y of point S s (y s =y e +WS10).

[0271] In addition, for example, depending on at least one of the location where the thick plate is placed, the position of the display area based on the template (label), and the direction in which the thick plate is placed, the mathematical formula representing the geometric positional relationship between the display area based on the template (label) and the first imaging device may not be used as a reference. Fig.18 as well as Fig.21Formula for explanation. For example, when a plurality of thick plates are stacked on a truck and a portion of the plurality of thick plates cannot be photographed by the camera, the height direction position of the thick plate placed on the top of the plurality of thick plates is sometimes uncertain. In addition, the mathematical formula representing the geometric positional relationship between the display area based on the template (label) and the first camera sometimes becomes complicated. Therefore, for example, the first camera may also capture a brightness image and a distance image as the first captured image. In addition, the brightness image is the captured image described so far in this embodiment. The brightness image may also have information on the intensity of the reflected light from the subject (brightness of the subject), and does not have information on the distance from the camera to the subject. The distance image has information on the distance from the camera to the subject. In addition, the first camera captures the brightness image and the distance image in such a way that the pixel position of the brightness image corresponds to the pixel position of the distance image. In addition, as a method for measuring the distance image, it is sufficient to use a known method such as a TOF (Time Of Flight) method or a pattern irradiation method.

[0272] In this case, the position derivation unit 1206 determines the pixels corresponding to the display area based on the template (label) based on the brightness image. For example, the position derivation unit 1206 pre-stores the image of the template (label) as a template image. The position derivation unit 1206 determines the display area based on the template (label) based on the brightness image by performing pattern matching between the brightness image and the template image. The position derivation unit 1206 determines the area in the area of ​​the distance image that corresponds to the display area based on the template (label) determined based on the brightness image. The position derivation unit 1206 extracts the pixel value of the determined area from the distance image. The pixel value of the distance image is a value representing the distance from the first camera to the subject. The pixel value extracted from the distance image in this way represents the distance from the first camera to the display area based on the template (label). In this way, the position derivation unit 1206 can obtain the distance from the first camera to the specified position of the display area based on the template (label) based on the pixel value extracted from the distance image. The position derivation unit 206 derives the three-dimensional position (xyz coordinates) of the specified position of the display area based on the template (label) based on the three-dimensional position (xyz coordinates) of the first camera device, the posture of the first camera device when capturing the first captured image, and the distance from the first camera device to the specified position of the display area based on the template (label).

[0273] <<Second Modification>>

[0274] As in the present embodiment, if the customer name 302, specification 303, size 304, ID 305, customer code 306, order number 307, specification 311, ID 312, size 313, delivery date 314, and customer code 315 are set as information that can be recognized by humans, it is more preferable because the inventory management by the operator (operator) becomes easy. However, at least one of the customer name 302, specification 303, size 304, ID 305, customer code 306, order number 307, specification 311, ID 312, size 313, delivery date 314, and customer code 315 may be converted (encoded) into information that cannot be recognized by humans, such as a barcode or a QR code.

[0275] <<Third Modification>>

[0276] In this embodiment, if Figure 3A , Figure 3B As shown, the case where the ID 305 displayed by the template is the same as the ID 312 displayed by the label is illustrated. However, the representation and recorded contents of these IDs are sometimes different. For example, there is also a form without a "hyphen" in the ID. In this case, information indicating that these IDs (template-based ID, label-based ID) are for the same thick plate is stored in the management table 1300. For display items other than ID, sometimes the representation and recorded contents of the same display item are different in the template-based display and the label-based display. For example, in the case where the specification is displayed in both the template and the label, for example, sometimes the abbreviation of the specification is displayed on the label. In addition, sometimes the number of digits of the needer code is different in the template-based display and the label-based display, but the needer code is managed by the manufacturer. In addition, for the size, sometimes not all the digits of the size are displayed. In this case, information indicating that these different representations and recorded contents are equivalent is also stored in the management table 1300.

[0277] <<Fourth Modification>>

[0278] In the present embodiment, the case where the three-dimensional position (xyz coordinates) of the display area based on the template (label) is derived as the three-dimensional position of the thick plate is illustrated. However, the three-dimensional position of the thick plate is not limited to the three-dimensional position (xyz coordinates) of the display area based on the template (label). For example, the item management device 1200 may also only identify one element displayed on the surface of the thick plate, and use the xyz coordinates of the position (such as the center of gravity or the centroid) of the identified element as the three-dimensional position of the thick plate. An element is information such as text, numbers, signs, and marks that can be processed as one information. For example, an element is information recognized as one information by the recognition unit 1202. The item management device 1200 may also, for example, recognize one element of the constituent elements of the display item in the display based on the template (label). In addition, the item management device 1200 may also recognize one element that is different from the constituent elements of the display item in the display based on the template (label). When the article management device 1200 derives the three-dimensional position of one element as the three-dimensional position of the thick plate, for example, it regards the circumscribed rectangle of the recognized one element as the display area 201 based on the template and performs the processing described in this embodiment.

[0279] In addition, the item management device 1200 may also only recognize two or more elements displayed on the surface of the thick plate. The two or more elements displayed on the surface of the thick plate are, for example, two or more constituent elements of the display items in the display based on the template (label). However, the two or more elements displayed on the surface of the thick plate are not all the constituent elements of the display items in the display based on the template (label). In addition, the two or more elements displayed on the surface of the thick plate may also be elements different from the constituent elements of the display items in the display based on the template (label). When the item management device 1200 derives the three-dimensional positions of the two or more elements as the three-dimensional position of the thick plate, for example, the circumscribed rectangle of the two or more elements displayed on the surface of the thick plate is regarded as the display area 201 based on the template and the processing described in this embodiment is performed.

[0280] Furthermore, even if the text is of the same size, the text located farther from the camera is photographed smaller than the text located closer to the camera. Using this, the position deriving unit 1206 can also derive the three-dimensional position of the object based on the position and size of the information captured in the first captured image.

[0281] <<Fifth Modification>>

[0282] In addition, the article management device 1200 may also include a direction deriving unit. The direction deriving unit derives information for determining the direction in which the article is placed based on a plurality of three-dimensional positions (xyz) on a surface of the article. Fig.14The three-dimensional positions (xyz coordinates) of the two vertices 1401a and 1401b of the circumscribed rectangle shown are used to derive the angle between the plate surface of the thick plate and the reference plane. The reference plane is, for example, the ground. A surface other than the ground can also be the reference plane. In this case, the angle between the plate surface of the thick plate and the reference plane is an example of information for determining the direction in which the object is placed. However, the information for determining the direction in which the object is placed is not limited to the angle between the plate surface of the object and the reference plane. For example, the symmetry axis (center axis) of the object can be used instead of the plate surface of the object.

[0283] In addition, the direction export unit can also replace Fig.14 The direction deriving unit may use the two vertices 1401a and 1401b of the circumscribed rectangle shown in the figure, or in addition thereto, the three-dimensional positions (xyz coordinates) of two or more components of the display item based on the template (label). In addition, the direction deriving unit may use the three-dimensional position (xyz coordinates) of information displayed in an area different from the display area based on the template (label) instead of these three-dimensional positions (xyz coordinates) or in addition thereto.

[0284] Furthermore, when this modification is adopted, Fig.13 In the non-stock display column of the management table 1300 shown in FIG. 1 , a column for storing information that determines the direction in which the item is placed is added. In the following description, the column that stores information that determines the direction in which the item is placed is referred to as an item direction column. The storage unit 1207 stores the information that determines the direction in which the thick plate is placed in the item direction column corresponding to the thick plate of the management table 1300. Thus, the information that determines the direction in which the thick plate is placed is stored in association with the ID and three-dimensional position of the thick plate.

[0285] In addition, when the plate surface of the thick plate is inclined relative to the ground, the position of the position (point) of the object to be derived of the three-dimensional position (xyz coordinates) in the height direction (z-axis direction) is derived, for example, as follows. First, the position deriving unit 1206 uses a laser rangefinder to derive the distance in the height direction (z-axis direction) from the first camera to the information (point) of the object to be derived of the three-dimensional position (xyz coordinates). Then, the position deriving unit 1206 derives the position obtained by subtracting the distance from the position of the first camera in the height direction (z-axis direction) as the position of the information (point) of the object to be derived of the three-dimensional position (xyz coordinates) in the height direction (z-axis direction).

[0286] Furthermore, as described in <<First Modification>>, the position deriving unit 1206 may derive the three-dimensional position (xyz coordinates) of the information (point) to be derived based on the luminance image and the range image.

[0287] <<Sixth Modification>>

[0288] In this embodiment, the case where the xyz coordinates of the display area based on the template (label) are derived as the three-dimensional position of the display area based on the template (label) is illustrated. However, the three-dimensional position of the display area based on the template (label) is not limited to the xyz coordinates. For example, the three-dimensional position of the display area based on the template (label) may also be a relative three-dimensional position based on the position of the first camera. In this case, the coordinates (xyz coordinates) of the point P representing the three-dimensional position of the camera may not be used. p ,y p ,z p ). In addition, when the origin of the xyz coordinate is used as the three-dimensional position of the first camera, the three-dimensional position of the display area based on the template (label) is represented by the xyz coordinate, which represents a relative three-dimensional position based on the three-dimensional position of the first camera.

[0289] <<7th Modification>>

[0290] In the present embodiment, the case where the identification information acquisition unit 1210 includes the identification unit 1202, the acquisition unit 1203, the comparison unit 1204, and the determination unit 1205 is illustrated. However, as long as the identification information acquisition unit 1210 acquires the identification information of the thick plate associated with the three-dimensional position derived by the position derivation unit 1206, it is not necessarily necessary to include the identification unit 1202, the acquisition unit 1203, the comparison unit 1204, and the determination unit 1205. The identification information acquisition unit 1210 may, for example, only include the identification unit 1202. In addition, the identification information acquisition unit 1210 may, for example, only include the acquisition unit 1203. In addition, the identification information acquisition unit 1210 may, for example, only include the identification unit 1202 and the acquisition unit 1203. For example, the identification information acquisition unit 1210 may also acquire the ID identified by the identification unit 1202 as the identification information of the thick plate. In this case, the identification information acquisition unit 1210 may not include the acquisition unit 1203, the comparison unit 1204, and the determination unit 1205. In addition, for example, when the object to be managed at the location is limited to one in advance, the identification information acquisition unit 1210 may also acquire the ID acquired by the acquisition unit 1203 as the identification information of the thick plate. In this case, the identification information acquisition unit 1210 may not include the identification unit 1202, the comparison unit 1204, and the determination unit 1205. In addition, for example, when the thick plate is determined only based on the information recognized by the identification unit 1202, the identification information acquisition unit 1210 may use the identification unit 1202, and when the object to be managed at the location is limited to one in advance, the acquisition unit 1203 may be used. In this case, the identification information acquisition unit 1210 may not include the comparison unit 1204 and the determination unit 1205 (it may only include the identification unit 1202 and the acquisition unit 1203).

[0291] <<Variant 8>>

[0292] In this embodiment, the case where the management table 1300 is stored in the item management device 1200 is exemplified. However, the management table 1300 may be located outside the item management device 1200.

[0293] <<Modification Example No. 9>>

[0294] In this embodiment, the case where the object whose position is changed is a thick plate is illustrated. However, the object whose position is changed is not limited to a thick plate. For example, the object whose position is changed may be a manufactured object other than a thick plate. The object whose position is changed may also be a manufactured object that displays information managed by a manufacturer. For example, a manufactured object may also be a billet, a steel billet, a thin plate coil, or a steel pipe. In addition, it may be an object other than a manufactured object. For example, the object whose position is changed may also be a container.

[0295] (Second embodiment)

[0296] Next, the second embodiment will be described. In the first embodiment, the case where the item management device 1200 manages the position of an item whose existing position has changed is illustrated. In this embodiment, the case where the item management device 1200 manages the position of an item installed in the installation area is described. Thus, in this embodiment, the item whose position is the object of management is not based on the premise that the existing position has changed. Thus, the main difference between this embodiment and the first embodiment lies in the different structure and processing depending on whether the item whose position is the object of management is based on the premise that the existing position has changed. Therefore, in the description of this embodiment, the same parts as those in the first embodiment are marked with the same reference numerals as in the first embodiment. Figure 1 to Figure 21 The same symbols as those in the figure are used and detailed description thereof will be omitted.

[0297] As items managed by the item management device 1200 of this embodiment, there are, for example, Fig. 22 as well as Fig.23 Items shown.

[0298] Fig. 22 FIG. 1 is a diagram showing an example of a steel frame of a structure installed on the ground. Fig. 22 In the example shown, the installation area is the ground. The structure is, for example, a building or a bridge. Fig. 22 In order to simplify the notation, only a part of the steel frame of the structure is simplified. Fig. 22 The following example is given: ID2202a~2202b is displayed on the surface of structural materials 2201a~2201b that serve as beam components among structural materials 2201a~2201d constituting a steel frame of a structure installed on the ground as information for uniquely identifying the structural materials 2201a~2201b. In addition, the structural material is, for example, a metal material such as steel. Structural materials 2201a~2201b are H-shaped steels. ID2202a~2202b is information corresponding to ID305 and 312. ID2202a~2202b is displayed, for example, by at least one of a template, handwriting, and engraving. In addition, ID2202a~2202b can also be displayed by a label. In addition, information other than ID2202a~2202b can also be displayed on the surface of structural materials 2201a~2201b.

[0299] In addition, Fig. 22In order to simplify the marking, the example in which ID 2202 is displayed only on structural material 2201a-2201b among structural materials 2201a-2201d is shown. However, information including information (ID) for uniquely identifying the structural material 2201c-2201d may also be displayed on the surface of the structural material 2201c-2201d. Fig. 22 In the example, the connection between the structural material 2201a and the structural materials 2201c to 2201d and the connection between the structural material 2201b and the structural materials 2201c to 2201d are respectively performed by welding. However, the connection method of the structural materials 2201a to 2201d is not limited to the method based on welding. For example, two structural materials can also be connected using a splicing plate.

[0300] exist Fig. 22 In the example shown, after the structural materials 2201a to 2201b are installed, before the surface of the structural materials 2201a to 2201b becomes invisible due to the coating of the structural materials 2201a to 2201b, a first camera is used to capture an image of the region of the structural materials 2201a to 2201b including the IDs 2202a to 2202b. In this way, the IDs 2202a to 2202b (information for uniquely identifying an item) are hidden and cannot be observed after installation. The function of the first captured image is the same as that of the first captured image described in the first embodiment. As the first camera, for example, a first camera held by an operator (operator) is used. In addition, as the first camera, a first camera provided by an unmanned aerial vehicle (drone) may also be used. In addition, in the following description, the region of the structural materials 2201a to 2201b including the IDs 2202a to 2202b is referred to as the ID display region of the structural material as needed. The ID display area of ​​the structural material corresponds to the display area based on the template (label) in the first embodiment.

[0301] Fig.23 FIG. 1 is a diagram showing an example of a pipe installed underground. Fig.23 In the example shown, the installation area is underground. The pipe is, for example, at least one of a metal pipe, an earth pipe, a vinyl chloride pipe, and an insulating pipe. Fig.23 2 shows an example in which ID 2302 is displayed on the surface of pipe 2301a among pipes 2301a to 2301c installed underground as information for uniquely identifying the pipe 2301a. ID 2302 is information corresponding to ID 305 and ID 312. ID 2302 is displayed, for example, by at least one of a template, handwriting, and engraving. In addition, ID 2302 may also be displayed by a label. In addition, as described in the first embodiment, information other than ID 2302 may also be displayed on the surface of pipe 2301a.

[0302] In addition, Fig.23 In order to simplify the marking, the case where ID 2302 is displayed only on tube 2301a among tubes 2301a to 2301c is illustrated. However, information including information (ID) for uniquely identifying the tubes 2301b to 2301c may also be displayed on the surface of the tubes 2301b to 2301c. Fig.23 2 illustrates an example in which the tubes 2301a to 2301c are connected using the joint members 2303a to 2303b. However, the method of connecting the tubes 2301a to 2301c is not limited to the method using the joint members 2303a to 2303b.

[0303] exist Fig.23 In the example shown, after the pipe 2301a is installed, before the pit 2304 is filled and the pipes 2301a to 2301c become invisible, a photographic image of the area of ​​the pipe 2301a including the ID 2302 is taken using the first camera. In this way, the ID 2302 (information for uniquely identifying an item) is hidden and cannot be observed after installation. The function of the first photographic image is the same as the function of the first photographic image described in the first embodiment. As the first camera, for example, a first camera held by an operator (operator) is used. In addition, as the first camera, a first camera provided by an unmanned aerial vehicle (drone) can also be used. In addition, in the following description, as needed, the area of ​​the pipe 2301a including the ID 2302 is referred to as the ID display area of ​​the pipe. The ID display area of ​​the pipe corresponds to the display area based on the template (label) of the first embodiment.

[0304] An example of the functional configuration of the article management device 1200 of this embodiment is similar to Fig.12 The example of the functional configuration of the article management device 1200 of the first embodiment shown is the same.

[0305] In addition, an example of the management table managed by the article management device 1200 of this embodiment is Fig.24The management table 2400 shown. In the following description including the drawings, the article management device 1200 (storage unit 1207) stores information for managing the positions of the structural materials (structural materials 2201a to 2201b, etc.) in the management table 2400. The article management device 1200 (storage unit 1207) does not need to store information for managing the positions of the pipes (pipe 2301a, etc.) in the management table 2400 when the article management device 1200 (storage unit 1207) manages the positions of the structural materials (structural materials 2201a to 2201b, etc.). On the other hand, the article management device 1200 (storage unit 1207) stores information for managing the positions of the pipes (pipe 2301a, etc.) in the management table 2400 when the article management device 1200 (storage unit 1207) manages the positions of the pipes (pipe 2301a, etc.). When the article management device 1200 (storage unit 1207) manages the position of the pipe (pipe 2301a, etc.), it is not necessary to store information for managing the position of the structural material (structural materials 2201a to 2201b, etc.) in the management table 2400. Fig.13 Same, in Fig.24 In the same column, the information also becomes identification information for the same structural material (tube) (the columns under the same No. become columns that store identification information for the same structural material (tube) respectively). In the following description, including the drawings, structural materials (structural materials 2201a to 2201b, etc.) and tubes (tube 2301a, etc.) are referred to as position-fixed items as needed.

[0306] A flowchart illustrating an example of processing of the article management device 1200 according to this embodiment is as follows: Fig.25 The flowchart for explaining an example of the processing of the article management device 1200 when managing the position of the structural material is different from the flowchart for explaining an example of the processing of the article management device 1200 when managing the position of the pipe. Fig.25 The flowchart shown is a flowchart for explaining an example of processing of the article management device 1200 when managing the position of the structural material. Fig.25 In the context of the present invention, "fixed-position articles" refer to "structural materials". Fig.25 The flowchart shown is a flowchart for explaining an example of the processing of the article management device 1200 when managing the position of the pipe. Fig.25 In the text, “fixed-position items” refer to “tubes”.

[0307] exist Fig. 22 as well as Fig.23 2 shows an example in which only IDs 2202a to 2202b and 2302 are displayed on structural members 2201a to 2201b and pipe 2301a. Fig.24In the management table 2400 shown, the stock display column only shows the ID column storing the IDs 2202a to 2202b and 2302. However, when information other than the IDs 2202a to 2202b and 2302 is displayed on the structural materials 2201a to 2201b and the pipe 2301a, the stock display column of the management table 2400 may include a column storing the information.

[0308] The information indicating the three-dimensional position of the position-fixed article is stored in the article position column of the management table 2400. The information stored in the article position column is information indicating the three-dimensional position of the position-fixed article determined by the article management device 1200. In this embodiment, the three-dimensional position of the ID display area of ​​the structural material and the three-dimensional position of the ID display area of ​​the pipe are respectively the three-dimensional position of the structural material and the three-dimensional position of the pipe. In addition, in this embodiment, the three-dimensional position of the ID display area of ​​the structural material and the three-dimensional position of the ID display area of ​​the pipe are examples of information display positions.

[0309] The registration time column of the management table 2400 stores information indicating the date and time (timing determined by the date and time) when the position-fixed item is placed at the current location. In addition, the date and time when the position-fixed item is placed at the current location refers to the date and time when the action for obtaining the date and time is performed. The registration time column stores, for example, the shooting date and time of the first captured image. However, the information stored in the registration time column is not limited to the shooting date and time of the first captured image (this is also the same in the first embodiment). For example, the operator can also input the information of the date and time when the position-fixed item is placed at the current location into the item management device 1200 by operating the user interface of the item management device 1200. In the present embodiment, the information stored in the registration time column is an example of the date and time when the action for obtaining the date and time when the item is placed at the current location is performed.

[0310] As described above, the ID2202a~2202b displayed on the structural material 2201a~2201b becomes invisible due to painting, etc. In addition, the ID2302 displayed on the pipe 2301a is buried underground and becomes invisible. Even in this case, the operator can grasp the elapsed time since the position-fixed item was installed based on the information stored in the registration time column. Therefore, for example, it is possible to judge whether the position-fixed item needs to be repaired or replaced based on the elapsed time since the position-fixed item was installed. In addition, based on the information stored in the ID column and the item position column, it is possible to grasp which position-fixed item exists in which position regardless of whether it is in a state where the ID2202a~2202b and 2302 can be visually confirmed. Therefore, it is possible to grasp which position-fixed item is located in which position of the position-fixed item that needs to be repaired or replaced. Therefore, the information stored in the management table 2400 can be used, for example, for the repair and replacement of the position-fixed item in a state where the ID (information for uniquely identifying the item) cannot be visually confirmed.

[0311] In addition, the non-stock display column of the management table 2400 may also include columns other than the item location column and the registration time column. For example, in the non-stock display column of the management table 2400, the item location column and the main ID column may be stored in the same manner as the management table 1300 of the first embodiment. In addition, in the first embodiment, an example is given of a case where the position within the item is a position with the specified position (vertices 203, 204) of the thick plate as the origin. A mark representing the origin may also be displayed at a specified position on the surface of the position-fixed item. The mark is displayed using, for example, paint. In addition, a straight line parallel to the axis of symmetry (center axis) of the position-fixed item may also be displayed on the surface of the position-fixed item. The straight line is displayed using, for example, paint. In this case, the first camera captures a first captured image including the ID display area of ​​the position-fixed item, the mark, and the straight line. In this case, the position with the position where the mark is displayed as the origin becomes the position within the item. In addition, the straight line and Fig.14 as well as Fig.15 In addition, the surface of the fixed object can also be displayed with Fig.14 The x-axis and Fig.15 The straight line corresponding to the z-axis shown. In addition, in the tube, the curvature corresponding to the outer diameter of the tube is considered to determine Fig.14 The x-axis and Fig.15 The distance from the origin on the z-axis is shown.

[0312] In addition, the item direction column described in the <<Fifth Modification>> of the first embodiment may also be included in the non-stock display column of the management table 2400. In addition, a tube has no plate surface. Therefore, for example, the angle between the symmetry axis (central axis) of the tube and the reference plane is stored in the item direction column. In this case, for example, multiple components (in Fig.23 In the example shown, ID 2302 is displayed on the surface of the tube 2301a in such a way that the direction in which the plurality of pixels are arranged (i.e., "4312") becomes a predetermined direction (e.g., a direction parallel to the axis of symmetry (central axis) of the tube 2301a).

[0313] Furthermore, when the ID of the fixed-position item and the three-dimensional position of the fixed-position item are determined based on the design drawing, the non-stock display column of the management table 2400 may also include an item assumed position column different from the item position. The three-dimensional position of the fixed-position item determined based on the design drawing is stored in the item assumed position column.

[0314] In the first embodiment, the first image acquisition unit 1201, the recognition unit 1202, the acquisition unit 1203, the comparison unit 1204, the determination unit 1205, the position derivation unit 1206, the storage unit 1207, the output unit 1208, and the second image acquisition unit 1209 perform processing using a display area based on a template (label). In contrast, in the present embodiment, the first image acquisition unit 1201, the recognition unit 1202, the acquisition unit 1203, the comparison unit 1204, the determination unit 1205, the position derivation unit 1206, the storage unit 1207, the output unit 1208, and the second image acquisition unit 1209 perform processing using an ID display area of ​​a structural material and an ID display area of ​​a pipe.

[0315] Therefore, in this embodiment, Fig.25 In step S2505, the identification unit 1205 identifies one position-fixed object from the position-fixed objects managed in the management table 2300 based on the comparison result of step S2504.

[0316] In addition, in this embodiment, the position deriving unit 1206 derives the three-dimensional position of the position-fixed object installed in the installation area. Therefore, for example, when it is pre-assumed which position-fixed object is installed at which position based on the design drawing, it is not necessary to perform Fig.25 In this case, the article management device 1200 may not include the second image acquisition unit 1209.

[0317] In addition, in step S2509, the position derivation unit 1206 may also derive the three-dimensional position of the position-fixed object using a mathematical formula representing the geometric positional relationship between the ID display area of ​​the position-fixed object and the first camera device, as in the first embodiment. In addition, the mathematical formula representing the geometric positional relationship between the ID display area of ​​the tube and the first camera device may also be represented by the surface curvature of the tube. In addition, the three-dimensional positional relationship between the ID display area of ​​the position-fixed object and the first camera device may also be derived using the information (size) of the ID display area of ​​the position-fixed object and the information (the number of pixels and size of the camera element, and the focal length) of the first camera device. In addition, as described in the first embodiment, when deriving the three-dimensional positional relationship between the ID display area of ​​the tube and the first camera device, the shape and size of the ID display area of ​​the tube and the surface curvature of the tube are used as the information of the ID display area of ​​the position-fixed object. In addition, the three-dimensional position of the position-fixed object may also be derived using a brightness image and a distance image.

[0318] Furthermore, the position derivation unit 1206 may determine whether the difference between the three-dimensional position of the position-fixed item derived as described above and the three-dimensional position stored in the item assumed position column of the non-stock display column of the management table is within a prescribed range. In this case, even if the difference between the three-dimensional position of the position-fixed item derived as described above and the three-dimensional position stored in the item assumed position column of the non-stock display column of the management table is not within a prescribed range, the processing of step S2512 may be performed.

[0319] In addition, the position derivation unit 1206 may also perform the following processing in step S2509. That is, the position derivation unit 1206 may also derive the three-dimensional position stored in the column corresponding to the position-fixed item determined in step S2505 in the assumed position column of the item in the non-spot display column of the management table as the three-dimensional position of the position-fixed item determined in step S2505. However, when the three-dimensional position of the position-fixed item is derived in this way, when the position-fixed item is not installed in the designed position, sometimes the accurate position cannot be derived as the three-dimensional position of the position-fixed item. Therefore, in the case of deriving the three-dimensional position of the position-fixed item in this way, the operator preferably confirms whether the position-fixed item is installed in the three-dimensional position based on the information indicating the three-dimensional position of the position-fixed item output in S2511.

[0320] In addition, in this embodiment, the position deriving unit 1206 derives the three-dimensional position of the position-fixed object installed in the installation area. Therefore, after the three-dimensional position of the position-fixed object is derived by the position deriving unit 1206 in step S2509, the three-dimensional position of the position-fixed object does not necessarily need to be updated. Fig.17The "update" shown in steps S1710 and S1713 is in Fig.25 This is indicated by the fact that "stored" is used in steps S2510 and S2513. However, after the three-dimensional position of the position-fixed object is derived by the position derivation unit 1206 in step S2509, the three-dimensional position of the position-fixed object may be updated. For example, based on the information indicating the three-dimensional position of the position-fixed object output in step S2511, if the three-dimensional position is obviously different from the actual three-dimensional position, the operator may instruct the item management device 1200 to re-derive the three-dimensional position of the position-fixed object. In this case, the execution is performed again. Fig.25 By processing the flowchart, the three-dimensional position of the fixed object can be updated in step S2510.

[0321] As described above, in the present embodiment, the item management device 1200 obtains a first captured image including a display area, the display area showing information displayed on a position-fixed item installed on the ground or underground, and an ID managed by the manufacturer of the position-fixed item. The item management device 1200 derives the three-dimensional position of the ID display area of ​​the position-fixed item based on the obtained first captured image and the position of the first imaging device that captured the first captured image, and stores the information representing the three-dimensional position in a storage medium in association with the ID of the position-fixed item. Therefore, for example, even in a case where the information displayed on the position-fixed item cannot be confirmed in reality after the position-fixed item is installed, it is possible to manage which position-fixed item is located at which position. Therefore, the position of the position-fixed item installed in the installed area can be appropriately managed.

[0322] In addition, the items installed in the installation area are not limited to Fig. 22 The structural materials shown and Fig.23 In addition, in this embodiment, various modification examples described in the first embodiment can also be adopted.

[0323] (Other Modifications)

[0324] In addition, the embodiments of the present invention described above can be implemented by executing a program on a computer. In addition, a computer-readable recording medium having the above program recorded thereon and a computer program product such as the above program can also be applied as an embodiment of the present invention. As a recording medium, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a non-volatile memory card, a ROM, etc. can be used. In addition, the embodiments of the present invention can be implemented by a PLC (Programmable Logic Controller) or by dedicated hardware such as an ASIC (Application Specific Integrated Circuit).

[0325] In addition, the embodiments of the present invention described above are only examples of specific implementation of the present invention, and the technical scope of the present invention is not limited thereto. That is, the present invention can be implemented in various ways without departing from its technical ideas or its main features.

[0326] (hardware)

[0327] An example of hardware of the article management device 1200 is described below. Fig.26 In the embodiment, the item management device 1200 includes a CPU 2601 , a main storage device 2602 , an auxiliary storage device 2603 , a communication circuit 2604 , a signal processing circuit 2605 , an image processing circuit 2606 , an I / F circuit 2607 , a user interface 2608 , a display 2609 , and a bus 2610 .

[0328] The CPU 2601 generally controls the entire article management device 1200. The CPU 2601 uses the main storage device 2602 as a work area and executes the program stored in the auxiliary storage device 2603. The main storage device 2602 temporarily stores data. The auxiliary storage device 2603 stores various data in addition to the program executed by the CPU 2601. In addition, a processor other than the CPU (for example, a GPU) may also be used.

[0329] The communication circuit 2604 is a circuit for communicating with the outside of the article management device 1200. The communication circuit 2604 can communicate with the outside of the article management device 1200 wirelessly or by wire.

[0330] The signal processing circuit 2605 performs various signal processing on the signal received from the communication circuit 2604 and the signal input under the control of the CPU 2601 .

[0331] The image processing circuit 2606 performs various image processing on the signal input under the control of the CPU 2601. The signal subjected to the image processing is input to the display 2609, for example.

[0332] The user interface 2608 is a part for the operator to instruct the article management device 1200. The user interface 2608 includes, for example, buttons, switches, and knobs. In addition, the user interface 2608 may include a graphical user interface using the display 2609.

[0333] The display 2609 displays an image based on the signal output from the image processing circuit 2606. The I / F circuit 2607 exchanges data with a device connected to the I / F circuit 2607. Fig.26 In the figure, a user interface 2608 and a display 2609 are shown as devices connected to the I / F circuit 2607. However, the devices connected to the I / F circuit 2607 are not limited thereto. For example, a removable storage medium may be connected to the I / F circuit 2607. In addition, at least a part of the user interface 2608 and the display 2609 may be located outside the item management device 1200.

[0334] The output unit 1208 is implemented by using, for example, the communication circuit 2604 and the signal processing circuit 2605 , and at least any one of the image processing circuit 2606 , the I / F circuit 2607 , and the display 2609 .

[0335] In addition, the CPU 2601, the main storage device 2602, the auxiliary storage device 2603, the signal processing circuit 2605, the image processing circuit 2606, and the I / F circuit 2607 are connected to the bus 2610. Communication between these components is performed via the bus 2610. In addition, the hardware of the article management device 1200 is not limited as long as it can realize the functions of the article management device 1200 described above. Fig.26 Hardware shown.

[0336] Industrial Applicability

[0337] The present invention can be used, for example, to manage the location of items.

Claims

1. An article management device for managing the location of articles, characterized in that: have: One or more image acquisition units that acquire a captured image of a region including display information displayed on the article; An identification information acquisition unit, which acquires the identification information of the above-mentioned item; a position deriving unit for deriving a three-dimensional position at which the display information is displayed, that is, an information display position, based on the captured image acquired by the image acquiring unit; and The storage unit stores the information display position in the article derived by the position deriving unit in association with the identification information of the article acquired by the identification information acquiring unit in a storage medium.

2. The article management device according to claim 1, characterized in that: The above-mentioned display information displayed on the above-mentioned object includes the identification information of the object, The identification information acquisition unit includes at least one of an identification unit and an acquisition unit. The recognition unit recognizes the identification information of the object based on the captured image acquired by the image acquisition unit. The acquisition unit acquires the identification information of the object stored in the storage unit.

3. The article management device according to claim 2, characterized in that: The information display position stored in the storage medium in association with the identification information of the object includes information related to a three-dimensional position based on a predetermined position of the object as information related to the three-dimensional position at which the display information is displayed. The captured image acquired by the image acquisition unit includes a region including a predetermined position of the article.

4. The article management device according to claim 2 or 3, characterized in that: The identification information acquisition unit includes: the identification unit mentioned above; The acquisition unit mentioned above; a comparing unit, which compares the identification information of the object identified by the identifying unit with the identification information of the object acquired by the acquiring unit; and The determining unit determines the identification information of the article included in the captured image acquired by the image acquiring unit based on the comparison result of the comparing unit.

5. The article management device according to any one of claims 1 to 4, characterized in that: The position deriving unit derives the position in the height direction of the surface of the article on which the display information is displayed, based on the two or more captured images acquired by the two or more image acquiring units.

6. The article management device according to any one of claims 1 to 5, characterized in that: The above-mentioned storage unit stores the above-mentioned information display position on the above-mentioned object exported by the above-mentioned position export unit as the current position of the above-mentioned object in the above-mentioned storage medium, and stores the date and time when the object is placed at the current position in association with the above-mentioned information display position on the object in the above-mentioned storage medium.

7. The article management device according to any one of claims 1 to 6, characterized in that: The article management device manages the location of articles whose locations have changed. The above-mentioned storage unit is such that, in a case where the above-mentioned information display position on the above-mentioned article and the identification information of the article have been stored in the above-mentioned storage medium in association with each other, and in a case where the above-mentioned information display position on the article is newly exported by the above-mentioned position export unit, the above-mentioned information display position already stored in the above-mentioned storage medium is updated to the above-mentioned information display position newly exported by the above-mentioned position export unit.

8. The article management device according to any one of claims 1 to 6, characterized in that: The article management device manages the positions of articles installed in the installation area.

9. A method for managing the location of items, characterized in that: include: One or more image acquisition steps of acquiring a captured image of a region including display information displayed on the article; an identification information obtaining step of obtaining the identification information of the above-mentioned article; a position derivation step of deriving a three-dimensional position where the display information is displayed, that is, an information display position, based on the captured image obtained in the image acquisition step; and A storing step of storing the information display position on the article derived by the position deriving step in association with the identification information of the article acquired by the identification information acquiring step in a storage medium.

10. A data structure for article management data, comprising: Display information displayed on the item; and Non-display information not displayed on the above items, The non-display information includes a three-dimensional position where the display information is displayed, that is, an information display position. The article management device is used to derive the information display position on the article based on the captured image of the area including the display information displayed on the article.

11. A program for causing a computer to execute a process for managing the location of an item, characterized in that: The above program causes the above computer to execute: One or more image acquisition steps of acquiring a captured image of a region including display information displayed on the article; an identification information obtaining step of obtaining the identification information of the above-mentioned article; a position derivation step of deriving a three-dimensional position where the display information is displayed, that is, an information display position, based on the captured image obtained in the image acquisition step; and A storing step of storing the information display position on the article derived by the position deriving step in association with the identification information of the article acquired by the identification information acquiring step in a storage medium.

Images