Information processing device and information processing program

Abstract

To easily monitor work by a worker.SOLUTION: An information processing device according to an embodiment includes a detection unit, a first determination unit, and a second determination unit. The detection unit detects a marker reflected in an image obtained by photographing a predetermined photographing space with a camera. The first determination unit determines a distance from the camera to the marker detected by the detection unit. The second determination unit considers the distance determined by the first determination unit and, when an object virtualized at a predetermined relative position with respect to the marker detected by the detection unit is in a predetermined positional relation with a predetermined arrangement space, determines that the object arranged in the arrangement space as a work object by a worker to which the marker detected by the detection unit is attached.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device and an information processing program. [Background technology]

[0002] 2. Description of the Related Art Various monitoring systems have been proposed for monitoring the work of workers on a production line or the like. However, the system is difficult to implement because it requires the installation of a large number of sensors and the execution of complex information processing. In view of these circumstances, it has been desirable to be able to easily monitor the work being performed by workers. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-560765 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present invention is to provide an information processing device and an information processing program that enable easy monitoring of work by a worker. [Means for solving the problem]

[0005] The information processing device according to the embodiment includes a detection unit, a first determination unit, and a second determination unit. The detection unit detects a marker captured in an image obtained by capturing a predetermined capturing space with a camera. The first determination unit determines a distance from the camera to the marker detected by the detection unit. The second determination unit determines, in consideration of the distance determined by the first determination unit, that an operation relating to an object placed in the placement space is an operation performed by a worker to which the marker detected by the detection unit is attached when a predetermined positional relationship is established between a virtual object at a predetermined relative position to the marker detected by the detection unit and the predetermined placement space. The second determination unit determines that the work relating to the part is work performed by the worker when a placement space in which the part is placed as an object and the object have a predetermined positional relationship. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a perspective view showing an external appearance of a monitoring system according to an embodiment. [Diagram 2] FIG. 2 is a block diagram showing a main circuit configuration of the information processing device shown in FIG. [Diagram 3] 3 is a diagram showing a schematic configuration of a data record included in the parts box database shown in FIG. 2. [Figure 4] 3 is a flowchart of a monitoring process performed by a processor shown in FIG. 2. [Diagram 5] 5A and 5B are diagrams for explaining a process of determining a virtual sphere. [Figure 6] 13A and 13B are diagrams for explaining modified examples of objects. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] An example of an embodiment will be described below with reference to the drawings. In this embodiment, a monitoring system that monitors the work of picking out parts from a parts box placed on a parts shelf in a predetermined order will be described as an example. FIG. 1 is a perspective view showing the appearance of a monitoring system 100 according to this embodiment.

[0008] The monitoring system 100 is provided on a parts shelf 200 . The component shelf 200 is a hollow rectangular parallelepiped with one side open. The component shelf 200 includes a plurality of shelf boards 201 and a support member 202. In the following, the open side of the component shelf 200 is referred to as the front side of the component shelf 200, and the direction of the front side as viewed from the component shelf 200 is referred to as the forward side. Shelf boards 201 are provided in the internal space of component shelf 200. The number and positions of shelf boards 201 are arbitrary, but FIG. 1 shows an example of three shelves. As shown in the figure, multiple component boxes 300 are arranged on shelf board 201. Each of the component boxes 300 contains components. The multiple component boxes 300 generally contain different components. The placement location of each of the component boxes 300 is predetermined. The support member 202 is gate-shaped and attached to the outer surface of the top plate of the component shelf 200 in a state where it protrudes upward.

[0009] The surveillance system 100 includes an information processing device 1, a camera unit 2, a display unit 3, and a speaker unit 4. The information processing device 1 executes information processing for monitoring a worker performing a predetermined task using parts stored in the parts box 300. The information processing device 1 is placed on the top plate of the parts shelf 200. However, the information processing device 1 may be placed at any position within the parts shelf 200, such as within the internal space below the parts shelf 200, or may be attached to the outside of a side wall of the parts shelf 200. Furthermore, the information processing device 1 may be placed at any location unrelated to the parts shelf 200, such as on the floor on which the parts shelf 200 is placed, or in a room other than the one in which the parts shelf 200 is installed.

[0010] The camera unit 2 is attached to the support member 202. The camera unit 2 captures an overhead image of a predetermined shooting area in front of the parts shelf 200. The camera unit 2 also measures the distance to an object located in the shooting area. The camera unit 2 includes a camera device suitable for measuring the distance to an object, such as a time of flight (TOF) camera or a stereo camera. Alternatively, the camera unit 2 may include a general camera device without a distance measurement function and a distance sensor separate from the camera device. The camera unit 2 may be installed in any manner as long as it can capture an image of the shooting area. For example, the camera unit 2 may be attached to the ceiling of a room in which the parts shelf 200 is installed.

[0011] The display unit 3 displays a screen for notifying the worker of various pieces of information. The display unit 3 may include various display devices, such as a liquid crystal display and a lamp. The display unit 3 is placed on the top plate of the parts shelf 200 in a state where the screen is easily visible to the worker. However, the display unit 3 may be attached to the outside of a side wall of the parts shelf 200. Furthermore, the display unit 3 may be installed in any location other than the parts shelf 200, such as by being attached to a wall of a room in which the parts shelf 200 is placed or hung from the ceiling of the same room.

[0012] Speaker unit 4 outputs various sounds to notify the worker of various pieces of information. Speaker unit 4 is attached to support member 202 in a state where the output sounds can be easily heard by the worker. However, speaker unit 4 may also be attached to the outside of a side wall of parts shelf 200. Furthermore, speaker unit 4 may be installed in any location other than parts shelf 200, such as by being attached to a wall of the room in which parts shelf 200 is placed or hung from the ceiling of the same room.

[0013] Fig. 2 is a block diagram showing a main circuit configuration of the information processing device 1. In Fig. 2, the same elements as those shown in Fig. 1 are denoted by the same reference numerals. The information processing device 1 includes a processor 11, a main memory 12, an auxiliary storage unit 13, an interface unit 14, and a transmission path 15.

[0014] The processor 11, the main memory 12, the auxiliary storage unit 13, and the interface unit 14 are capable of communicating with each other via a transmission path 15. The processor 11, the main memory 12, and the auxiliary storage unit 13 are connected by the transmission path 15 to configure a computer for controlling the information processing device 1.

[0015] The processor 11 corresponds to the central part of the computer. The processor 11 executes information processing for realizing various functions of the information processing device 1 according to an operating system and information processing programs such as application programs. The processor 11 is, for example, a CPU (central processing unit).

[0016] The main memory 12 corresponds to the main storage portion of the computer. The main memory 12 includes a nonvolatile memory area and a volatile memory area. The main memory 12 stores the information processing program in the nonvolatile memory area. The main memory 12 may store data required for the processor 11 to execute information processing in a nonvolatile or volatile memory area. The main memory 12 uses the volatile memory area as a work area where data is appropriately rewritten by the processor 11. The nonvolatile memory area is, for example, a ROM (read only memory). The volatile memory area is, for example, a RAM (random access memory).

[0017] The auxiliary storage unit 13 corresponds to the auxiliary storage portion of the computer. As the auxiliary storage unit 13, for example, a storage unit using a well-known storage device such as an EEPROM (electric erasable programmable read-only memory), a HDD (hard disk drive), or an SSD (solid state drive) can be used. The auxiliary storage unit 13 stores data used by the processor 11 in performing various processes, or data created by the processes in the processor 11. The auxiliary storage unit 13 may also store the above-mentioned information processing program.

[0018] The camera unit 2, the display unit 3, and the speaker unit 4 are respectively connected to the interface unit 14. The interface unit 14 interfaces data transmission between the processor 11 and the camera unit 2, the display unit 3, and the speaker unit 4. As the interface unit 14, well-known devices such as various interface boards, for example, a USB (universal serial bus) board, can be used. A plurality of interface units 14 may be provided. The transmission path 15 includes an address bus, a data bus, and control signal lines, and transmits data and control signals between the connected components.

[0019] The auxiliary storage unit 13 stores a monitoring program PRA, which is one of the information processing programs. The monitoring program PRA is an application program in which information processing, described below, for implementing the functions of the information processing device 1 is described. A part of the storage area of ​​the auxiliary storage unit 13 is used as a storage area for a parts box database DAA. The parts box database DAA is a database for managing each of the multiple parts boxes 300.

[0020] FIG. 3 is a diagram showing a schematic configuration of a data record REA contained in the parts box database DAA. The parts box database DAA is a collection of multiple data records REA associated with multiple parts boxes 300 respectively. The data record REA includes fields FEA, FEB, and FEC. The field FEA is set with a box number as an identifier of the associated parts box 300. The field FEB is set with area data representing the part placement space for the associated parts box 300. The field FEC is set with a part code as an identifier of the part contained in the associated parts box 300.

[0021] For example, a general-purpose computer device can be used as the hardware of the information processing device 1. The information processing device 1 is generally transferred in a state where the hardware in a state where the monitoring program PRA is not stored in the auxiliary storage unit 13 is separate from the monitoring program PRA. In this case, the information processing device 1 is configured by writing the monitoring program PRA to the auxiliary storage unit 13 in response to an operation by an arbitrary worker. The information processing device 1 may be transferred in a state where the monitoring program PRA is stored in the auxiliary storage unit 13. The monitoring program PRA may be transferred by recording it on a removable recording medium such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory, or by communication via a network. The monitoring program PRA may be stored in the main memory 12.

[0022] Next, the operation of the monitoring system 100 configured as above will be described. A worker is positioned in front of the parts shelf 200, and takes out parts from the parts box 300 in a predetermined order, and performs work such as assembling a product using the parts. A box number is assigned to each of the multiple parts boxes 300, and the size of the parts box 300, which parts are stored in which parts box 300, and the positions of each of the parts boxes 300 on the parts shelf 200 are determined in advance. The parts box database DAA is then appropriately determined according to the predetermined environment, for example, by an administrator of the monitoring system. The arrangement space represented by the area data set in the field FEB of the data record REA is determined as an area corresponding to the storage space of the parts inside the associated parts box 300, for example. The area data is determined as data representing which area of ​​the arrangement space is within the parts shelf 200 and its surrounding three-dimensional space. However, the area data may be any data that can specify the rough location of the storage location of the parts, such as data representing which area the parts box 300 occupies in the three-dimensional space.

[0023] The worker performs work while wearing a band on his / her wrist, to which a marker is attached that will be reflected in an image captured by the camera unit 2. The marker is rectangular and configured such that a first corner, a second corner, and a third corner out of four corners can be identified by image processing. The worker wears the band so that the first corner and the second corner of the marker are located closer to the hand than the third corner and another corner.

[0024] When the monitoring system 100 is in a state of monitoring the work of a worker, the camera unit 2 captures an image of the shooting area. Typically, the image is captured by shooting a moving image, but it may also be captured by repeatedly shooting still images. The processor 11 in the information processing device 1 starts information processing based on the surveillance program PRA (hereinafter referred to as surveillance processing) every time a predetermined start timing arrives. The start timing is assumed to be, for example, a timing for each frame rate of the camera unit 2. However, the start timing may be arbitrarily determined by the creator of the surveillance program PRA or the administrator of the surveillance system 100, etc. 4 is a flowchart of the monitoring process by the processor 11. Note that the contents of the process described below are one example, and the order of some of the processes can be changed, some of the processes can be omitted, or other processes can be added as appropriate.

[0025] As ACT11, the processor 11 acquires output data from the camera unit 2. When shooting a moving image, the camera unit 2 shoots at a predetermined frame rate and outputs the resulting frame data. The camera unit 2 also measures the distance to an object present in the shooting area for multiple measurement points within the frame area and outputs the resulting measurement data. The processor 11 then acquires the latest frame data and measurement data output by the camera unit 2. The processor 11 stores the acquired frame data and measurement data in the main memory 12 or the auxiliary storage unit 13.

[0026] As ACT12, the processor 11 detects a marker reflected in an image represented by the frame data acquired in ACT11. The processor 11 attempts to extract an area in which the characteristics of the marker appear from the image. If a marker is reflected in the image, the processor 11 detects the area in which the marker is reflected as the marker. Thus, the processor 11 executes information processing based on the monitoring program PRA, and the computer with the processor 11 as the central part functions as a detection unit.

[0027] In ACT 13, the processor 11 checks whether or not the marker has been successfully detected. If the marker has been detected in ACT 12, the processor 11 judges the result as YES and proceeds to ACT 14. As ACT14, the processor 11 determines a spherical area within the parts shelf 200 and the three-dimensional area around it (hereinafter referred to as the monitoring area). That is, the processor 11 imagines a sphere that exists within the monitoring area. Hereinafter, this sphere will be referred to as a virtual sphere. The virtual sphere is an example of a imagined object.

[0028] FIG. 5 is a diagram for explaining the process of determining the virtual sphere. 5 shows an example of an image in which a worker's hand, a belt BEA worn by the worker, and a marker MAA attached to the belt BEA are captured. However, the worker's hand and the belt BEA are shown for reference and are not objects to be processed by the processor 11.

[0029] An example of a specific process performed by the processor 11 to determine the virtual sphere is as follows. First, the processor 11 obtains the positions POA, POB, and POC of the first corner, second corner, and third corner of the marker MAA in the image. Then, the processor 11 determines the direction of the position POA based on the position POC as the hand direction. That is, the processor 11 determines the direction of the arrow ARA in FIG. 5 as the hand direction. Next, the processor 11 determines the position POE in the horizontal plane at the center of the virtual sphere as a position away from the position POD, which is the midpoint between the positions POA and POB, by a specified distance DIA in the hand direction. The processor 11 determines the distance from the camera unit 2 to the marker MAA (hereinafter referred to as the shooting distance) based on the measurement data acquired in ACT11. For example, the processor 11 selects a measurement value related to a measurement point in the area where the marker MAA is detected from the measurement values ​​of multiple measurement points shown in the measurement data acquired in ACT11, and determines the distance represented by the measurement value as the shooting distance. Thus, the processor 11 executes information processing based on the monitoring program PRA, so that the computer having the processor 11 as its central part functions as a first determination unit.

[0030] Then, the processor 11 changes the prescribed distance DIA according to the determined shooting distance, for example. The relationship between the shooting distance and the prescribed distance DIA is determined in advance, for example, by the creator of the surveillance program PRA, so that the larger the shooting distance, the smaller the prescribed distance DIA. However, the processor 11 may calculate the prescribed distance DIA according to the shooting distance using a formula determined in advance by the creator of the surveillance program PRA, for example, taking into account the optical characteristics of the camera unit 2, etc. Furthermore, the processor 11 determines a three-dimensional position of the center based on the position POE and the shooting distance, and determines a sphere centered on the position as the virtual sphere SPA. The processor 11 changes the size of the virtual sphere SPA according to the shooting distance. The relationship between the shooting distance and the size of the virtual sphere SPA is determined in advance, for example, by the creator of the surveillance program PRA, so that the larger the shooting distance, the smaller the virtual sphere SPA. However, the processor 11 may calculate the size of the virtual sphere SPA according to the shooting distance using a formula determined in advance by the creator of the surveillance program PRA, for example, taking into account the optical characteristics of the camera unit 2, etc.

[0031] In ACT15, the processor 11 checks whether the virtual sphere SPA is in contact with any of the arrangement spaces of the parts box 300. For this check, a well-known algorithm for determining collision between objects can be used. For example, the processor 11 selects one of the data records REA included in the parts box database DAA, and checks whether a part of the virtual sphere SPA overlaps with the arrangement space represented by the area data set in the field FEB. The processor 11 then repeats the above check, for example, by sequentially changing the selected data record REA until it checks whether there is an overlap, and if it checks whether there is an overlap, it judges that there is contact and gives YES, and proceeds to ACT16.

[0032] As ACT16, the processor 11 determines the work performed by the worker. For example, the processor 11 determines the work performed by the worker as work using a part identified by a part code set in the field FEC of the data record REA that was selected when it was confirmed that a part of the virtual sphere SPA overlapped. Thus, the processor 11 executes information processing based on the monitoring program PRA, and the computer with the processor 11 as the central part functions as a second determination unit.

[0033] In ACT17, the processor 11 checks whether the work determined as above is currently a regular work. For example, the processor 11 refers to history data that records the work performed by the worker over time and schedule data that indicates the schedule of the work that the worker should perform to determine the work that the worker should perform at the current time, and if the work does not match the work determined in ACT16, it determines that the work is not regular work and judges it as NO, and proceeds to ACT18. However, the conditions under which the processor 11 determines that the work is regular may be appropriately determined by, for example, the creator of the monitoring program PRA. The history data and schedule data are stored in, for example, the main memory 12 or the auxiliary storage unit 13. However, the history data and schedule data may be stored in a storage device provided in another server device that is capable of communicating via a communication network such as a LAN (local area network).

[0034] As ACT18, the processor 11 executes an alarm action. For example, the processor 11 displays a predetermined alarm screen on the display unit 3. For example, the processor 11 blinks a lamp provided on the display unit 3. For example, the processor 11 outputs a predetermined alarm sound from the speaker unit 4. For example, the processor 11 outputs a predetermined voice message from the speaker unit 4. The processor 11 may execute various other actions that can cause the operator to perceive the alarm as the alarm action. The processor 11 may execute only one of these alarm actions, or may execute multiple actions simultaneously or at different times. Then, if the termination condition is met, the processor 11 stops the alarm action and proceeds to ACT19. The termination condition may be appropriately determined, for example, by the creator of the monitoring program PRA or the administrator of the monitoring system 100. It is assumed that the termination condition is, for example, when the duration of the alarm action reaches a specified time. It is assumed that the termination condition is, for example, when a predetermined action is performed by the operator.

[0035] If the processor 11 cannot detect the marker in ACT12, it judges NO in ACT13 and proceeds to ACT19. If the processor 11 cannot confirm any overlap even after selecting all of the data records REA included in the parts box database DAA in ACT15, it judges NO as there is no contact and proceeds to ACT19. If the work that the worker should currently be performing matches the work judged in ACT16 in ACT17, it judges YES as it is a regular work and proceeds to ACT19. In these cases, the processor 11 skips ACT18 and does not perform an alarm operation. In ACT 19, the processor 11 updates the history data to add data representing the current action of the worker, and then ends the monitoring process.

[0036] As described above, according to the information processing device 1, it is possible to monitor work by imagining an object based on a captured image and confirming the positional relationship between the object and the arrangement space, which can be realized by simple information processing.

[0037] Furthermore, according to the information processing device 1, the object is a virtual sphere SPA that is a spherical area, so that the information processing for determining the object is simple, while the general position of the worker's hands can be grasped to monitor the work.

[0038] Furthermore, according to the information processing device 1, when contact between an object and the arrangement space is detected, it is determined that the work to be performed by the worker involves using a part that is arranged in the arrangement space, which can be achieved with simpler information processing than when the work is determined based on the actual behavior of the worker involved in the work.

[0039] This embodiment can be modified in various ways as follows. The virtual object is not limited to a sphere, but may be an object of any other shape.

[0040] FIG. 6 is a diagram for explaining a modified example regarding the virtualization of an object. In this modified example, rays RAA and RAB are virtual objects. Ray RAA is virtualized as a straight line of a specified length that has one end at the position POD and extends in the hand tip direction. Ray RAB is virtualized as a straight line of a specified length that has one end at the position POD and extends in a direction with a predetermined inclination with respect to the hand tip direction. The processor 11 may then determine the operation of the worker based on, for example, the contact state between the respective tips of the rays RAA and RAB and the arrangement space.

[0041] The processor 11 can appropriately change the predetermined positional relationship, for example, determining that the predetermined positional relationship has been achieved when the distance of the virtual sphere SPA to the configuration space becomes smaller than a predetermined distance.

[0042] The process for determining that an object and a configuration space have a predetermined positional relationship can be modified as appropriate, such as associating a plane with a configuration space through which a hand passes when inserting a hand into the configuration space, and processor 11 determining that work related to a part placed in the configuration space with which the plane is associated is the worker's work depending on contact between the plane and an object.

[0043] The processor 11 can appropriately change the work to be determined, such as determining that work using the tool in question is work performed by a worker when the object has a predetermined positional relationship with a placement space defined as the space in which the tool is placed.

[0044] Each function realized by the processor 11 through information processing can be realized in part or in whole by hardware that executes information processing not based on a program, such as a logic circuit, etc. Also, each of the above functions can be realized by combining the above hardware, such as the logic circuit, with software control.

[0045] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. The invention as originally claimed in the present application is set forth below. [Appendix 1] A detection unit that detects a marker reflected in an image obtained by photographing a predetermined photographing space with a camera; a first determination unit that determines a distance from the camera to the marker detected by the detection unit; a second determination unit that, when a predetermined positional relationship is established between an object imaginary at a predetermined relative position with respect to the marker detected by the detection unit and a predetermined placement space, determines that an operation relating to the object placed in the placement space is an operation performed by a worker to which the marker detected by the detection unit is attached, in consideration of the distance determined by the first determination unit; An information processing device comprising: [Supplementary Note 2] The second determination unit imagines the object as a spherical region having a center position determined by the position of the marker detected by the detection unit and a size according to the distance determined by the first determination unit. 2. An information processing device according to claim 1. [Appendix 3] The second determination unit detects contact between the object and the arrangement space, and determines that a predetermined positional relationship has been established upon detection of the contact. 3. An information processing device according to claim 1 or 2. [Additional Note 4] The second determination unit determines that the work on the part is a work performed by a worker when the arrangement space in which the part is arranged as an object and the object have a predetermined positional relationship. The information processing device according to any one of Supplementary Note 1 to Supplementary Note 3. [Appendix 5] Computer, A detection unit that detects a marker captured in an image obtained by capturing an image of a predetermined capturing space with a camera; a first determination unit that determines a distance from the camera to the marker detected by the detection unit; a second determination unit that, when a predetermined positional relationship is established between an object imaginary at a predetermined relative position with respect to the marker detected by the detection unit and a predetermined placement space, determines that the object placed in the placement space is a work target to be performed by a worker to which the marker detected by the detection unit is attached, in consideration of the distance determined by the first determination unit; An information processing program that enables the system to function as such. [Explanation of symbols]

[0046] 1...information processing device, 2...camera unit, 3...display unit, 4...speaker unit, 11...processor, 12...main memory, 13...auxiliary storage unit, 14...interface unit, 15...transmission path, 100...monitoring system, 200...parts shelf, 201...shelf board, 202...support member, 300...parts box.

Claims

1. A detection unit that detects a marker captured in an image obtained by capturing an image of a predetermined capturing space with a camera; a first determination unit that determines a distance from the camera to the marker detected by the detection unit; a second determination unit that, when a predetermined positional relationship is established between a virtual object at a predetermined relative position to the marker detected by the detection unit and a predetermined placement space, determines that an operation relating to the object placed in the placement space is an operation performed by a worker to which the marker detected by the detection unit is attached, taking into consideration the distance determined by the first determination unit; Equipped with the second determination unit determines that the work on the part is work performed by a worker when the arrangement space in which a part is arranged as an object and the object have the predetermined positional relationship. Information processing device.

2. the second determination unit imagines the object as a spherical region having a center position determined by the position of the marker detected by the detection unit and a size according to the distance determined by the first determination unit; The information processing device according to claim 1 .

3. the second determination unit detects contact between the object and the arrangement space, and determines that the predetermined positional relationship has been achieved upon detection of the contact.

3. The information processing device according to claim 1 or 2.

4. Computer, A detection unit that detects a marker captured in an image obtained by capturing an image of a predetermined capturing space with a camera; a first determination unit that determines a distance from the camera to the marker detected by the detection unit; a second determination unit that, when a predetermined positional relationship is established between an object imaginary at a predetermined relative position with respect to the marker detected by the detection unit and a predetermined placement space, determines that the object placed in the placement space is a work target to be performed by a worker to which the marker detected by the detection unit is attached, in consideration of the distance determined by the first determination unit; and make it work. the second determination unit determines that a task relating to the part is a task performed by a worker when the arrangement space in which a part is arranged as an object and the object are in the predetermined positional relationship; Information processing program.

Images