Image processing device, image processing method, information processing system,

Abstract

To improve operational stability in a system that uses image processing results for images.SOLUTION: An image processing device 100 of the present invention comprises first processing means 121 for performing first processing for recognizing a first object in an image, identification means 122 for identifying power consumed for the first processing, and determination means 123 for determining whether or not to perform second processing for recognizing a second object in the image according to the identified power consumption.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and an information processing system. [Background technology]

[0002] The use of technology for detecting objects from image data in various fields is being considered. For example, Patent Document 1 describes the detection of moving objects such as people from image data captured by a surveillance camera installed in a store.

[0003] Here, the above-mentioned surveillance cameras can be installed on the ceiling or walls inside a store. For example, Patent Document 2 describes that power is supplied to a network camera and an image processing device via PoE (Power over Ethernet). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-237884 [Patent Document 2] Japanese Patent Application Publication No. 2018-148454 Summary of the Invention [Problem to be solved by the invention]

[0005] On the other hand, when powering objects installed on the ceiling or walls of a store, the amount of power that can be supplied is limited. For this reason, Patent Document 2 changes the processing status according to the power consumption of the network camera and image processing device. In particular, Patent Document 2 describes that image recognition processing is stopped in a low power consumption operation mode.

[0006] However, as described in Patent Document 2, if image recognition processing is stopped depending on power consumption, the desired image processing results cannot be obtained. Therefore, in a system that executes a predetermined process using the image processing results for an image, the problem of reduced operational stability arises. Furthermore, the same problem as above occurs in systems that use any power supply method, not just PoE power supply.

[0007] SUMMARY OF THE INVENTION Therefore, an object of the present invention is to improve the operational stability of a system that uses the results of image processing on an image. [Means for solving the problem]

[0008] An image processing device according to one aspect of the present invention includes: a first processing means for performing a first process of recognizing a first object from an image; A determination unit that determines the power consumption required for the first process; a determination means for determining whether to perform a second process of recognizing a second object from the image according to the identified power consumption; Equipped with The structure is as follows.

[0009] Furthermore, an image processing method according to one aspect of the present invention includes: performing a first process of recognizing a first object from the image; Identifying power consumption required for the first process; determining whether to perform a second process of recognizing a second object from the image according to the identified power consumption; The structure is as follows.

[0010] Furthermore, an information processing system according to one aspect of the present invention includes: a first processing means for performing a first process of recognizing a first object from an image; A determination unit that determines the power consumption required for the first process; a determination means for determining whether to perform a second process of recognizing a second object from the image according to the identified power consumption; Equipped with The structure is as follows.

[0011] Furthermore, a program according to one aspect of the present invention includes: performing a first process of recognizing a first object from the image; Identifying power consumption required for the first process; determining whether to perform a second process of recognizing a second object from the image according to the identified power consumption; Have the computer perform the process, The structure is as follows. [Effects of the Invention]

[0012] With the above-described configuration, the present invention can improve the operational stability of a system that uses the results of image processing on an image. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a hardware configuration of an image processing apparatus according to a first embodiment of the present invention. [Figure 2] 1 is a block diagram showing a configuration of an image processing device according to a first embodiment of the present invention. [Figure 3] 3 is a flowchart showing the operation of the image processing device according to the first embodiment of the present invention. [Figure 4] FIG. 10 is a schematic diagram showing the configuration of an information processing system according to a second embodiment of the present invention. [Figure 5] FIG. 5 is a block diagram showing the configuration of the image processing device and the control device disclosed in FIG. 4. [Figure 6] FIG. 6 is a diagram showing the state of processing by the image processing device disclosed in FIG. 5. [Figure 7] FIG. 6 is a diagram showing the state of processing by the image processing device disclosed in FIG. 5. [Figure 8] 6 is a flowchart showing the operation of the image processing device disclosed in FIG. 5. [Figure 9] FIG. 10 is a block diagram showing the configuration of an image processing device according to a third embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing the state of processing by the image processing device disclosed in FIG. 9. DETAILED DESCRIPTION OF THE INVENTION

[0014] <Embodiment 1> Next, a first embodiment of the present invention will be described with reference to Fig. 1 to Fig. 3. Fig. 1 and Fig. 2 are block diagrams showing the configuration of an image processing device in the first embodiment, and Fig. 3 is a flowchart showing the operation of the image processing device. Note that this embodiment shows an outline of the configuration of an image processing device, an information processing system, and an image processing method that will be described in the following embodiments.

[0015] First, the hardware configuration of an image processing device 100 according to this embodiment will be described with reference to Fig. 1. The image processing device 100 is configured as a general information processing device, and is equipped with the following hardware configuration, for example. ·CPU(Central Processing Unit)101(Arithmetic unit) ROM (Read Only Memory) 102 (storage device) RAM (Random Access Memory) 103 (storage device) Programs 104 loaded into RAM 103 A storage device 105 for storing a group of programs 104 A drive device 106 that reads and writes from a storage medium 110 external to the information processing device A communication interface 107 that connects to a communication network 111 outside the information processing device Input / output interface 108 for inputting and outputting data Bus 109 connecting each component

[0016] The image processing device 100 can be equipped with the first processing means 121, the identification means 122, and the determination means 123 shown in Fig. 2 by having the CPU 101 acquire and execute the program group 104. The program group 104 is stored in advance in the storage device 105 or the ROM 102, for example, and is loaded into the RAM 103 and executed by the CPU 101 as needed. The program group 104 may be supplied to the CPU 101 via the communication network 111, or may be stored in advance in the storage medium 110, and the drive device 106 may read out the program and supply it to the CPU 101. However, the first processing means 121, the identification means 122, and the determination means 123 described above may be constructed using dedicated electronic circuits for realizing such means.

[0017] Note that FIG. 1 shows an example of the hardware configuration of an information processing device, which is the image processing device 100, and the hardware configuration of the information processing device is not limited to the above-described case. For example, the information processing device may be configured with only a part of the above-described configuration, such as excluding the drive device 106. Furthermore, the image processing device 100 may be configured with multiple information processing devices. The above-described first processing means 121, identification means 122, and determination means 123 do not necessarily have to be installed in a single information processing device. For example, each of the means 121, 122, and 123 itself or each processing function of each means may be distributed and installed in multiple information processing devices. In other words, an information processing system including multiple information processing devices may include the above-described means 121, 122, and 123.

[0018] The image processing device 100 then executes the image processing method shown in the flowchart of FIG. 3 by the functions of the first processing means 121, the specifying means 122, and the determining means 123, which are constructed by the program as described above.

[0019] As shown in FIG. 3, the image processing device 100 The first processing means 121 performs a first process of recognizing a first object from an image (step S101), The determination means 122 determines the power consumption required for the first process (step S102), The determination means 123 determines whether to perform a second process of recognizing a second object from the image according to the identified power consumption (step S103). The following process is executed.

[0020] Here, the image refers to an image captured of a predetermined space where a recognition target may exist. For example, the image may be any image captured by any imaging device, such as a depth camera, stereo camera, or 3D LiDAR (Light Detection and Ranging) camera, or an RGB image captured by a spectral camera. The image may also be a combination of multiple types of images. The first target includes any moving or stationary object, such as a robot, luggage, person, or vehicle. The first process refers to a process of detecting the first target from an image when the first target is included in the image. For example, the first process detects the first target by detecting its characteristics (distance, shape, color, etc.) from the image, but the first target may be detected by any process.

[0021] Furthermore, the power consumption required for the first process is the power consumed by the image processing device when the image processing device executes the first process. The amount of power consumed by the first process may be the power consumption of the image processing device at the end of the first process. "Specifying the power consumption" refers to acquiring the value of the power consumption, for example, by acquiring a value detected by a power detection device or acquiring the power consumption calculated from the processing content. "Depending on the specified power consumption" refers to depending on the value of the specified power consumption, for example, when the power consumption value is less than a set threshold. The threshold may be a preset value or a value set according to the power consumption value.

[0022] The second target includes any moving or stationary object, such as a robot, luggage, person, or vehicle, and is an object different from the first target. The second process refers to a process of detecting the second target from an image when the second target is included in the image. For example, the second process detects the second target by detecting features (distance, shape, color, etc.) that indicate that the second target is a different object from the first target from the image, but any process may be used to detect the second target. The term "determining whether to perform the second process" refers to a decision to perform the second process depending on the value of the power consumption described above.

[0023] In the present invention, with the above-described configuration, first, the first processing means 121 of the image processing device 10 detects whether a first object is included in an image captured by the imaging device 130. Then, if the first object is included in the image, the first processing means 121 executes a first process to detect the first object from the image. Next, the identification means 122 of the image processing device 100 detects the power consumption required for the first process, that is, the value of the power consumed by the image processing device 100 itself in the process of detecting the first object from the image. Then, the determination means 123 of the image processing device 100 determines whether the detected power consumption value satisfies a preset criterion. If the power consumption satisfies the preset criterion, the image processing device 100 decides to execute a second process to further detect a second object from the image.

[0024] In this way, in the present invention, since the second object is recognized according to the power consumption required for recognizing the first object, it is possible to improve the operational stability of a system that executes predetermined processing using image processing results. Specifically, since the second object is recognized according to the power consumption required for recognizing the first object, that is, the second object is further recognized when there is a margin in power consumption, it is possible to reduce the possibility of failure in recognition of the control object due to insufficient power, and it is possible to stably operate a system that executes predetermined processing using image processing results.

[0025] <Embodiment 2> A second embodiment of the present invention will be described with reference to Figures 4 to 8. Figures 4 to 5 are diagrams for explaining the configuration of an information processing system, and Figures 6 to 8 are diagrams for explaining the processing operation of the information processing system.

[0026] [composition] The information processing system of the present invention is configured to include an imaging device C that images a predetermined space, and an image processing device 10 that processes the captured image. The predetermined space to be imaged is, for example, a warehouse in which an object to be transported is placed, and is a space where an unmanned transport robot R transports the object to be transported. Therefore, the predetermined space is a space where the transport robot R travels, and the information processing system is configured to control the transport robot R as a control object.

[0027] As shown in FIG. 4, the image capturing device C is installed on a ceiling W or a wall of a warehouse or the like, and is configured to capture an image of an area in which a transport robot R to be controlled moves. The image processing device 10 processes the captured image to detect the position of the transport robot R that moves to transport items within the warehouse, and also detects the positions of objects other than the transport robot R, such as placed luggage T1 and people T2 that may hinder the movement of the transport robot R. The positions of the transport robot R and obstacles T1 and T2 detected by the image processing device 10 are notified to the control device 20 and are used to control the movement of the transport robot R. Although the image processing device 10 and the control device 20 are shown as separate devices in FIG. 4, the image processing device 10 and the control device 20 may be configured as a single information processing device.

[0028] However, the information processing system of the present invention is not necessarily limited to being installed in a warehouse, but may be installed in any location. For example, it may be installed in a factory, such as a production line where an industrial robot such as a robot arm is installed, or outdoors where there are many people and devices. Furthermore, the objects detected by the image processing device 10 are not limited to the robot R or obstacles T1 and T2, but any object may be detected. The information processing system may be used for any purpose. The configuration of the information processing system will be described in detail below.

[0029] The camera device C is installed on the ceiling W inside the warehouse and photographs a predetermined area inside the warehouse from above. However, the camera device C is not necessarily limited to being installed on the ceiling W, but may be installed anywhere, such as on a wall or a stand. Therefore, the image captured by the camera device C is not limited to being an image seen from above, but may be an image captured from any direction.

[0030] The image capturing device C in this embodiment is a depth camera that captures distance images in which pixel values ​​represent distance values ​​(depth). For example, the image capturing device C may be a 3D LiDAR that measures distance from the round-trip time when an infrared laser is projected onto an object, or may be any type of image capturing device, such as a stereo camera that captures distance images using parallax. The image capturing device C captures images at regular time intervals and passes the images to the image processing device 10. However, the image capturing device C is not limited to a depth camera and may be any camera, such as a spectral camera that captures RGB images, or may be a combination of multiple types of cameras. Furthermore, the images captured by the image capturing device C are not limited to distance images and may be any image, such as an RGB image, or may be a combination of multiple types of images. In this embodiment, the image capturing device C captures and acquires two types of images: distance images and RGB images.

[0031] The image processing device 10 is composed of one or more information processing devices each having a calculation device and a storage device. In particular, the image processing device 10 in this embodiment is configured integrally with the above-mentioned image capture device C and is installed on the ceiling W of the warehouse together with the image capture device C. For example, the image processing device 10 is configured as a single-board computer so that it can be installed on the ceiling W, and since it is integrated with the image capture device C, it is configured as a so-called network camera. However, the image processing device 10 is not necessarily limited to being configured integrally with the image capture device C, and may be configured separately from the image capture device C. Furthermore, the image processing device 10 is not necessarily limited to being installed on the ceiling W, and may be installed anywhere, such as on a wall or a rack.

[0032] As shown in FIG. 5 , the image processing device 10 includes a network power receiving unit 15, and can receive power via a communication network. Specifically, by connecting a LAN (Local Area Network) cable to the network power receiving unit 15 of the image processing device 10, power can be received via PoE (Power over Ethernet). Note that, although this varies depending on the length and quality of the LAN cable, there is an upper limit to the power that can be supplied to the image processing device 10. Therefore, the image processing device 10 operates on power supplied via PoE with an upper limit value set for power consumption. Note that the photographing device C configured integrally with the image processing device 10 also operates on power supplied to the image processing device 10 via PoE. However, the image processing device 10 is not necessarily limited to receiving power via PoE, and may receive power via any method.

[0033] As shown in FIG. 5, the image processing device 10 includes an image acquisition unit 11, an object detection unit 12, a power management unit 13, and an information generation unit 14. The functions of the image acquisition unit 11, the object detection unit 12, the power management unit 13, and the information generation unit 14 can be realized by a computing device executing a program for realizing each function stored in a storage device. The image processing device 100 may be configured with multiple information processing devices. The image acquisition unit 11, the object detection unit 12, the power management unit 13, and the information generation unit 14 described above do not necessarily have to be installed in a single information processing device. For example, the units 11 to 14 themselves or the processing functions of each unit may be distributed and installed in multiple information processing devices. In other words, an information processing system configured with multiple information processing devices may include the units 11 to 14. Each component will be described in detail below.

[0034] The image acquisition unit 11 acquires images, which are distance images and RGB images, captured by the photographing device C and temporarily stores them in a storage device. At this time, since the photographing device C captures images at regular time intervals, the image acquisition unit 11 sequentially acquires images and stores them in the storage device.

[0035] The object detection unit 12 (first processing means, second processing means, third processing means, determination means) performs a process of recognizing an object from the acquired image. Specifically, the object detection unit 12 first performs a process (first process) of recognizing a transport robot R (first object) that may be present in the image using the distance image. At this time, the object detection unit 12 constantly performs the process of recognizing the transport robot R at time intervals of, for example, several tens of milliseconds to 100 milliseconds.

[0036] As an example, the object detection unit 12 stores the height of the top of the head of the transport robot R in advance, and recognizes an object at the stored height as the transport robot R from within the image, which is a distance image. For example, the top of the head of the transport robot R is formed by a plane with a predetermined shape (for example, a rectangle), and the height of the plane is predetermined. Therefore, the object detection unit 12 can recognize an object at the stored height as the transport robot R by recognizing the object. At this time, the object detection unit 12 may recognize the transport robot R by taking into consideration the outer shape of the object. Note that the object detection unit 12 may recognize the transport robot R from within the image by any method. For example, the object detection unit 12 may recognize the transport robot R from within the image using an RGB image. As an example, a QR code containing identification information for identifying the transport robot R may be displayed on the top surface of the transport robot R, and the transport robot R may be recognized by reading the QR code from the RGB image. Then, the object detection unit 12 passes the coordinates of the recognized transport robot R in the image to the information generation unit 14. Note that reference symbol A1 in FIG. 7 indicates the state of the above-mentioned processing, and as indicated by reference symbol P1, the transport robot R is recognized from within the image.

[0037] The object detection unit 12 also performs a process (second process) to recognize objects other than the transport robot that may be present in the image, such as a baggage T1 or a person T2 that may be an obstacle as shown in FIG. 4. At this time, the object detection unit 12 performs the obstacle recognition process at time intervals of, for example, several hundred milliseconds to 1 second, i.e., at longer time intervals than the recognition process for the transport robot R described above. However, the object detection unit 12 performs the obstacle recognition process in accordance with the power consumption of the image processing device 10 in operation. For example, the power consumption of the image processing device 10 in operation is identified by the power management unit 13 (described later). If the identified power consumption is less than a set threshold, the object detection unit 12 determines to perform the obstacle recognition process and executes the obstacle recognition process. On the other hand, if the identified power consumption is equal to or greater than the set threshold, the object detection unit 12 does not perform the obstacle recognition process. The threshold to be compared with the power consumption at this time is a first threshold, which is set in advance to a value lower than the upper limit of power consumption set for the image processing device 10.

[0038] Specifically, the object detection unit 12 performs a process of recognizing an obstacle as follows. In this embodiment, the object detection unit 12 uses a distance image and an RGB image in combination to recognize an obstacle from within the image. However, the object detection unit 12 is not necessarily limited to using a distance image and an RGB image in combination, and may use either one of the images or the other image.

[0039] First, the object detection unit 12 sets a lattice-shaped grid G ​​in the image as shown by the dotted lines in Fig. 7. For example, the intersection O of the orthogonal solid lines shown in Fig. 7 represents the center position of the image capture device C, and an array of square grid G ​​is set around this center position. The size of the grid G ​​and its reference point are notified by the grid setting unit 21 of the control device 20, as will be described later.

[0040] The object detection unit 12 then recognizes areas at heights (depths) other than those of the transport robot R from the distance image and recognizes each such area as an obstacle area. For example, the object detection unit 12 recognizes areas at heights (depths) that are considered to be the same as the height of the transport robot R as a single obstacle area. In this case, the object detection unit 12 may capture an initial image in advance when no obstacles exist, and recognize the obstacle area by calculating the difference between the initial image and a newly acquired image. Alternatively, the object detection unit 12 may recognize the obstacle area by performing the above-described difference processing using an RGB image in addition to the distance image. The object detection unit 12 then determines whether each grid G ​​set as described above corresponds to an obstacle area. In this case, the object detection unit 12 determines that an obstacle is present for a grid G ​​whose coordinates correspond to the obstacle area, and determines that no obstacle is present for a grid G ​​whose coordinates correspond to a non-obstacle area. For example, in the example of FIG. 7 , the grid indicated by the symbol U1, which is filled in black, is determined to have an obstacle. On the other hand, if the object detection unit 12 is located around the obstacle area but is unable to determine that it is an obstacle area through the above-described recognition process due to the shadow of an object in the RGB image, the object detection unit 12 determines that it is an unknown obstacle. For example, in the example of FIG. 7 , the grid indicated by the symbol U2, which is filled in gray, is determined to have an unknown obstacle. The object detection unit 12 then notifies the information generation unit 14 of the determination results for each grid G. For example, the object detection unit 12 notifies the information generation unit 14 of the position information of each grid G, indicating U1 with an obstacle and U2 with an unknown obstacle. Note that the position information of the grid G ​​may be, for example, information on the placement position of the grid G ​​in the image, or the coordinates of the vertices of the grid G ​​in the image, or any information that can identify the position in the image.

[0041] Here, the manner in which the above-mentioned obstacles T1 and T2 are recognized is shown by reference symbol A2 in Fig. 6. When the obstacles T1 and T2 are recognized as described above, the transport robot R within the frame indicated by reference symbol P1 is recognized from the image as shown in the diagram indicated by reference symbol A2, and the obstacles T1 and T2 within the frame indicated by reference symbol P2 are also recognized.

[0042] In the above description, the object detection unit 12 recognizes obstacles by setting grids G in the image and detecting obstacles for each grid G, but any method may be used to recognize obstacles from an image. For example, the object detection unit 12 may detect, as an obstacle, an object that does not have the characteristics (for example, height or shape) of the transport robot R from a range image or an RGB image, and recognize the obstacle by detecting the coordinates of the obstacle in the image.

[0043] Furthermore, the object detection unit 12 performs further processing (third processing) on ​​the image of the obstacle recognized as described above. Here, the object detection unit 12 performs processing to determine the type of the recognized obstacle. At this time, the object detection unit 12 performs the processing to determine the type of obstacle at a time interval of, for example, 1 second or more, that is, at a time interval longer than the time interval for the obstacle recognition processing described above. However, the object detection unit 12 performs the processing to recognize the obstacle according to the power consumption of the image processing device 10 in operation. For example, the power consumption of the image processing device 10 in operation is identified by the power management unit 13 described below. If the identified power consumption is less than a set threshold, the object detection unit 12 determines to perform the processing to determine the type of obstacle, and performs the processing to determine the type of obstacle. On the other hand, if the identified power consumption is equal to or greater than the set threshold, the object detection unit 12 does not perform the processing to determine the type of obstacle. Note that the threshold to be compared with the power consumption at this time is a second threshold, which is set in advance to a value lower than the upper limit of power consumption set for the image processing device 10.

[0044] The object detection unit 12 executes a process for determining the type of obstacle depending on the result of the process for recognizing the obstacle. For example, when the object detection unit 12 recognizes obstacles as described above, it identifies the number of such obstacles. Then, the object detection unit 12 notifies the power management unit 13 of the identified number of obstacles. As described below, the power management unit 13 then estimates the power consumption of the process for determining the type of the recognized obstacle, which will be performed later by the image processing device 10, and executes the process for determining the type of obstacle depending on the estimated power consumption. For example, if the estimated power consumption is less than a set threshold, the object detection unit 12 determines to execute the process for determining the type of obstacle, and executes the process for determining the type of obstacle. On the other hand, if the estimated power consumption is equal to or greater than the set threshold, the object detection unit 12 does not execute the process for determining the type of obstacle. Note that the threshold to be compared with the estimated power consumption at this time is a third threshold, which is a preset value lower than the upper limit of power consumption set for the image processing device 10.

[0045] Specifically, the object detection unit 12 performs a process of determining the type of obstacle as follows. First, the object detection unit 12 identifies the shape of each obstacle region in which an obstacle is recognized, and performs pattern matching between the shape and the shape of a known object. For example, pattern data of the shapes of luggage and people is prepared as known objects. Then, the object detection unit 12 determines that the obstacle in each obstacle region is an object whose shape matches the obstacle region, such as luggage T1 or person T2. ​​Then, the object detection unit 12 notifies the information generation unit 14 of the determination result of the type of each obstacle. However, the process of determining the type of obstacle by the object detection unit 12 is not limited to the above-described method, and may be performed by any method. Note that reference symbol A3 in FIG. 6 indicates the above-described process. As shown in the boxes marked with reference symbols P1 and P2, the transport robot R and obstacles T1 and T2 are recognized from the image, and as shown in the box marked with reference symbol P3, the types of the obstacles T1 and T2 are also determined from the image.

[0046] The power management unit 13 (identification means, estimation means) measures and identifies the power consumption of the image processing device 10 during operation, and notifies the object detection unit 12 of the identified power consumption. For example, when the object detection unit 12 of the image processing device 10 is performing a recognition process for the transport robot R, the power management unit 13 identifies the power consumption of the image processing device 10 to include the power consumption due to such process, and when the object detection unit 12 of the image processing device 10 is performing a recognition process for an obstacle, the power management unit 13 identifies the power consumption of the image processing device 10 to include the power consumption due to such process. Furthermore, when the power management unit 13 is notified of the number of recognized obstacles by the object detection unit 12 of the image processing device 10, the power management unit 13 estimates the power consumption when the image processing device 10 identifies the types of the number of obstacles, and notifies the object detection unit 12 of the estimated power consumption. For example, a calculation formula is provided for calculating the power consumption when the type of obstacle is identified depending on the number, and the power management unit 13 estimates the power consumption using such calculation formula. The reason for estimating power consumption here is that the power consumption for subsequent processes, such as the process of determining the type of obstacle, varies depending on the number of obstacles, and this is used to determine whether to perform such subsequent processes.

[0047] The information generation unit 14 generates position information of the transport robot R and the obstacles T1 and T2 based on the information notified by the object detection unit 12 as described above, and notifies the control device 20 of these. For example, the information generation unit 14 notifies the control device 20 of position information consisting of coordinates on an image in which the transport robot R is recognized, or notifies the control device 20 of position information of a grid G ​​that has been determined to have an obstacle U1 or an unknown obstacle U2 as a result of recognizing the obstacles T1 and T2. The information generation unit 14 also notifies the control device 20 of the type of obstacle that has been determined.

[0048] The control device 20 is composed of one or more information processing devices each having a calculation device and a storage device. As shown in Fig. 5, the control device 20 has a communication unit 25, which is connected to the image processing device 10 via a communication device such as a hub. The communication unit 25 can also be connected to the transport robot R via a wireless communication device, and can transmit signals for transport control, as will be described later.

[0049] 5, the control device 20 includes a grid setting unit 21 and a control unit 22. The functions of the grid setting unit 21 and the control unit 22 can be realized by the calculation device executing a program for realizing each function stored in the storage device. Each component will be described in detail below.

[0050] The grid setting unit 21 notifies the image processing device 10 of setting information of the grid G ​​in the image that is set by the object detection unit 12 as described above. The setting information of the grid G ​​includes, for example, the size of the grid G ​​and its reference point.

[0051] The control unit 22 controls the movement of the transport robot G within the warehouse. Specifically, the control unit 22 first forms and manages a map of the area where transport is performed by the transport robot R, using the position information of the transport robot and the types and position information of the obstacles T1 and T2 notified from the information generation unit 14 of the image processing device 10 as described above. Then, the control unit 22 performs control such as setting a movement route for the transport robot R and instructing the transport robot R to move along the movement route based on the map.

[0052] [Operation] Next, the operation of the above-described information processing system, in particular the operation of the image processing device 10, will be described mainly with reference to the flowchart of FIG.

[0053] An upper limit of power consumption is set in advance in the image processing device 10. This upper limit of power consumption is set, for example, according to the power level that can be supplied via PoE power supply. Furthermore, several power consumption thresholds that are smaller than the upper limit are set in the image processing device 10. For example, as described above, a first threshold is set to be compared with the power consumption at the time when determining whether to perform the obstacle recognition process, a second threshold is set to be compared with the power consumption at the time when determining whether to perform the obstacle type discrimination process, and a third threshold is set to be compared with the power consumption estimated according to the number of obstacles in the obstacle type discrimination process. Furthermore, the time intervals for performing the image acquisition process, the obstacle recognition process, and the obstacle type discrimination process are also set in advance.

[0054] First, the image processing device 10 receives grid setting information from the grid setting unit 21 of the control device 20 (step S1). Then, the image processing device 10 sequentially acquires images captured by the imaging device C (step S2), performs processing to recognize the transport robot R from the images (step S3), and transmits position information of the transport robot R in the images to the control device 20 (step S4).

[0055] If the timing does not occur at a predetermined interval, such as a time interval of several hundred milliseconds to one second, i.e., if the timing is not a timing set for performing obstacle recognition processing, the image processing device 10 returns to the processing for acquiring the subsequent image. That is, the processing of steps S5 and S6 in FIG. 8 is not performed, and the processing returns to step S2. The image processing device 10 then performs recognition processing of the transfer robot R for the acquired subsequent image (step S3) and transmits position information (step S4). On the other hand, if the timing corresponds to the timing occurring at the above interval, the image processing device 10 calculates and identifies the power consumption of the image processing device itself, including the power consumption associated with the recognition processing of the transfer robot R (step S5), and checks whether the power consumption is less than a first threshold (step S6). If the identified power consumption is equal to or greater than the first threshold (No in step S6), the image processing device 10 does not perform the obstacle recognition processing, and proceeds to recognition of the transfer robot R for the subsequent image (step S2). On the other hand, if the identified power consumption is less than the first threshold (Yes in step S6), the image processing device 10 executes a process of recognizing an obstacle (step S7). That is, since the image processing device 10 has a margin of power consumption, the image processing device 10 executes a process of recognizing objects other than the transport robot R, that is, objects T1 and T2 that may become obstacles.

[0056] The image processing device 10 then performs a process of recognizing obstacles T1 and T2 from the image (step S7) and transmits position information of the grids in which the obstacles T1 and T2 are located in the image to the control device 20 (step S8). Furthermore, if the timing does not occur at a predetermined cycle, such as a time interval of 1 second or more, that is, if the timing is not a set timing for performing a process for determining the type of obstacle, the image processing device 10 does not perform the processes of steps S9 and S10 in FIG. 8 and returns to step S2, where the image processing device 10 proceeds to recognize the transport robot R in the subsequent image. On the other hand, if the timing corresponds to the cycle, the image processing device 10 calculates and identifies the power consumption of the image processing device itself, including the power consumption associated with the recognition process of the obstacles T1 and T2 (step S9), and checks whether the power consumption is less than a second threshold (step S10). If the identified power consumption is equal to or greater than the second threshold (No in step S10), the image processing device 10 proceeds to recognize the transport robot R in the subsequent image (step S2). On the other hand, if the identified power consumption is less than the second threshold (Yes in step S10), the image processing device 10 proceeds to a process of determining the type of the recognized obstacle (step S11). In other words, since the image processing device 10 has a margin in power consumption, it further performs a process of determining the types of the obstacles T1 and T2.

[0057] In step S10, the image processing device 10 may estimate the power consumption of the process of determining the type of the recognized obstacles based on the number of recognized obstacles T1 and T2. Then, the image processing device 10 may execute the process of determining the type of obstacles when the estimated power consumption is less than a third threshold.

[0058] Then, the image processing device 10 performs a process of determining the types of the obstacles T1 and T2 recognized from the image (step S11), and transmits obstacle information including the types of the obstacles T1 and T2 to the control device 20 (step S12). After that, the image processing device 10 proceeds to recognize the transport robot R in the subsequent image (step S2), and repeats the above-mentioned process.

[0059] The control device 20 acquires the position information of the transport robot and the type and position information of the obstacles T1 and T2 through the processing of the image processing device 10 described above, and uses this information to form and manage a map of the area where transport is performed by the transport robot R. Then, the control device 20 controls the transport operation, such as setting a movement route for the transport robot R and instructing the transport robot R on the movement route, based on the map.

[0060] As described above, according to the image processing device 10 of this embodiment, first, the transfer robot R is constantly recognized for the acquired image, and if there is a margin for power consumption at that time, the obstacles T1 and T2 are further recognized, and if there is still a margin for power consumption, the types of the obstacles T1 and T2 are determined. In this way, the image processing device 10 performs image processing such as recognition processing on the image in stages depending on the power consumption situation, so that it is possible to perform necessary image processing and acquire necessary information while suppressing excessive power consumption. As a result, it is possible to suppress the stop of image processing due to a power shortage, and it is possible to improve the operational stability of the image processing device 10.

[0061] In particular, in the case where the image processing device 10 forms a network camera and receives power via PoE, as in this embodiment, the system can operate stably while performing the necessary image processing even in a situation where the amount of power that can be supplied is limited.

[0062] Specifically, in this embodiment, the transport robot R is recognized from an image captured by a camera C installed on the ceiling or the like, and the obstacles T1 and T2 are detected in advance. Therefore, when controlling the movement of the transport robot R, the obstacles T1 and T2 can be smoothly avoided.

[0063] In the above description, the object detection unit 12 performs the obstacle recognition process at time intervals of several hundred milliseconds to one second and at time intervals of one second or more. However, the set values ​​of these time intervals may be changed depending on the power consumption. For example, the object detection unit 12 may change at least one of the time interval for the obstacle recognition process and the time interval for the obstacle type determination process depending on the power consumption value of the transport robot R during the recognition process. For example, the time interval may be set to be longer if the power consumption is equal to or greater than a predetermined threshold. Furthermore, for example, the object detection unit 12 may change the time interval for the obstacle type determination process depending on the power consumption value during the obstacle recognition process. For example, the time interval may be set to be longer if the power consumption is equal to or greater than a predetermined threshold. Furthermore, for example, the object detection unit 12 may change the time interval for the obstacle type determination process depending on the power consumption value estimated in the obstacle type determination process. For example, the time interval may be set to be longer if the power consumption is equal to or greater than a predetermined threshold.

[0064] <Embodiment 3> Next, a third embodiment of the present invention will be described with reference to Fig. 9 and Fig. 10. Fig. 9 is a diagram for explaining the configuration of an image processing device in the third embodiment, and Fig. 10 is a diagram for explaining the processing operation of the image processing device.

[0065] In addition to the configuration shown in Fig. 5 described in the first embodiment, the image processing device 10 of this embodiment further includes a log management unit 16 as shown in Fig. 9. The log management unit 16 is realized by the arithmetic device executing a program. The following mainly describes the configuration that is different from the second embodiment.

[0066] The log management unit 16 (detection means) stores a log of malfunctions occurring in the image processing device 10 and calculates the severity of the malfunction. Specifically, the log management unit 16 stores a log of malfunctions of preset items occurring in the hardware and software installed in the image processing device 10. As an example, a malfunction occurring in the hardware is when the operation of the image processing device 10 stops, and a malfunction occurring in the software is when processing by the software stops or when the software terminates. However, malfunctions occurring in the image processing device 10 are not limited to the above-mentioned contents.

[0067] The log management unit 16 calculates an error rate representing the probability of occurrence based on the number of times a malfunction has occurred for each of the above-mentioned hardware and software. For example, the log management unit 16 calculates the error rate at a predetermined time interval, such as every hour, and plots the error rate in an error rate table, as shown in FIG. 10, with the hardware error rate on the vertical axis and the software error rate on the horizontal axis. In this case, as shown by the dotted line, the error rate table sets a target value for each error rate (e.g., 0.1%) and thresholds lower and higher than the target value, and sets target regions and domains 1 to 6 divided by these values. Note that the target region in the error rate table in FIG. 10 is a region in which the error rate of both the hardware and software is below the target value and above the lower threshold.

[0068] The power management unit 13 (setting means) in this embodiment has a function of changing power consumption-related settings, such as the upper limit of power consumption set for the image processing device 10, the image acquisition cycle, the cycle of each image processing operation, and the first, second, and third thresholds, according to the current error rate plotted in the error rate table. For example, if the current error rate falls within the target range, the power management unit 13 leaves the current settings unchanged. On the other hand, if the current error rate falls within domain 1, the power management unit 13 changes the settings to lower power consumption because the error rate is high and stable operation is not being performed. For example, the power management unit 13 lowers the upper limit of power consumption, which is set in stages, by two stages, increases the image acquisition cycle to the maximum value, and minimizes each threshold. Note that each threshold may also be set in multiple stages, and the settings may be changed to thresholds set according to the domain to which the error rate belongs. Similarly, if the image processing device 10 belongs to domains 2 to 4, the error rate is still high and stable operation is not being performed, so the power consumption is changed to lower power consumption. For example, if the image processing device 10 belongs to domain 2, the upper limit of power consumption is lowered by one stage, the image acquisition cycle is increased by one stage, and each threshold is lowered by one stage. If the device belongs to domain 3, the upper limit of power consumption is lowered by one level, and if it belongs to domain 4, the image acquisition cycle is raised by one level and each threshold is lowered by one level. On the other hand, if the device belongs to domains 5 and 6, the error rate is lower than the allowed value, which means that the power resources are not being fully utilized, so the power consumption is changed to increase. For example, if the device belongs to domain 5, the image acquisition cycle is lowered by one level and each threshold is raised by one level. If it belongs to domain 6, the upper limit of power consumption is raised by one level.

[0069] As described above, according to this embodiment, the image processing device 10 can be set to an operating state that allows for efficient use of power while realizing stable operation according to the operating status of the image processing device 10, particularly the occurrence of errors.

[0070] Although the present invention has been described above with reference to the above-described embodiments, the present invention is not limited to the above-described embodiments. Various modifications that are understandable to those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. Furthermore, at least one or more of the above-described functions may be executed by an information processing device installed and connected anywhere on a network, i.e., may be executed by so-called cloud computing. Furthermore, each of the above-described functions is not necessarily limited to being installed in a single information processing device, but may be installed in multiple separate information processing devices.

[0071] The above-described program can be stored and supplied to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may also be supplied to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can supply the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0072] <Additional Notes> The above-described embodiments may be partially or entirely described as follows: The following provides an overview of the configurations of an image processing device, an image processing method, an information processing system, and a program according to the present invention. However, the present invention is not limited to the following configurations. (Appendix 1) a first processing means for performing a first process of recognizing a first object from an image; A determination unit that determines the power consumption required for the first process; a determination means for determining whether to perform a second process of recognizing a second object from the image according to the identified power consumption; An image processing device comprising: (Appendix 2) 10. The image processing device according to claim 1, the specifying means specifies the power consumption required for the second process; the determining means determines whether to execute a third process of determining the type of the second object from the image, depending on the power consumption required for the identified second process. Image processing device. (Appendix 3) 3. The image processing device according to claim 2, an estimation means for estimating power consumption required for the third processing based on a result of the second processing; the determining means determines whether to execute the third process depending on the estimated power consumption. Image processing device. (Appendix 4) 4. The image processing device according to claim 3, the estimation means estimates the power consumption required for the third process based on the number of the second objects recognized from the image. Image processing device. (Appendix 5) 5. An image processing device according to any one of Supplementary Notes 2 to 4, the determining means changes the timing of performing the second process or the third process depending on the identified power consumption required for the first process. Image processing device. (Appendix 6) 5. The image processing device according to claim 3, the determining means changes the timing of performing the third process in accordance with the estimated power consumption. Image processing device. (Appendix 7) 7. An image processing device according to any one of claims 1 to 6, a detection means for detecting the degree of a malfunction occurring in the image processing device; a setting means for setting a threshold value according to the degree of the defect; the determination means determines whether the second object is to be recognized from the image according to the power consumption and the threshold value. Image processing device. (Appendix 8) performing a first process of recognizing a first object from the image; Identifying power consumption required for the first process; determining whether to perform a second process of recognizing a second object from the image according to the identified power consumption; Image processing methods. (Appendix 9) 9. The image processing method according to claim 8, further comprising: Identifying the power consumption required for the second process; determining whether to execute a third process of identifying the type of the second object from the image according to the identified power consumption required for the second process; Image processing methods. (Appendix 10) 10. The image processing method according to claim 9, further comprising: estimating power consumption required for the third process based on a result of the second process; determining whether to execute the third process according to the estimated power consumption; Image processing methods. (Appendix 11) 11. The image processing method according to claim 10, further comprising: estimating power consumption required for the third process based on the number of the second objects recognized from the image; Image processing methods. (Appendix 12) 12. An image processing method according to any one of Supplementary Notes 9 to 11, changing a preset timing for performing the second process or the third process according to the identified power consumption required for the first process; Image processing methods. (Appendix 13) 12. The image processing method according to claim 10 or 11, changing a preset timing for performing the third process according to the estimated power consumption; Image processing methods. (Appendix 14) 14. An image processing method according to any one of Supplementary Notes 8 to 13, comprising: Detect the degree of the malfunction that has occurred in the image processing device, A threshold value is set according to the degree of the defect, determining whether to recognize the second object from the image according to the power consumption and the threshold value; Image processing methods. (Appendix 15) a first processing means for performing a first process of recognizing a first object from an image; A determination unit that determines the power consumption required for the first process; a determination means for determining whether a second object is to be recognized from the image in accordance with the identified power consumption; An information processing system comprising: (Appendix 16) 16. The information processing system of claim 15, the specifying means specifies the power consumption required for the second process; the determining means determines whether to perform a second process of executing a third process of determining the type of the second object from the image, depending on the power consumption required for the specified second process. Information processing system. (Appendix 17) 17. The information processing system of claim 16, an estimation means for estimating power consumption required for the third processing based on a result of the second processing; the determining means determines whether to execute the third process depending on the estimated power consumption. Information processing system. (Appendix 18) 18. The information processing system of claim 17, the estimation means estimates the power consumption required for the third process based on the number of the second objects recognized from the image. Information processing system. (Appendix 19) 19. An information processing system according to any one of Supplementary Notes 16 to 18, the determining means changes the timing of performing the second process or the third process depending on the identified power consumption required for the first process. Information processing system. (Appendix 20) 19. The information processing system according to claim 17, the determining means changes the timing of performing the third process in accordance with the estimated power consumption. Information processing system. (Appendix 21) 21. The information processing system according to any one of Supplementary Notes 15 to 20, a detection means for detecting the degree of a malfunction occurring in the information processing system; a setting means for setting a threshold value according to the degree of the defect; the determination means determines whether the second object is to be recognized from the image according to the power consumption and the threshold value. Information processing system. (Appendix 22) performing a first process of recognizing a first object from the image; Identifying power consumption required for the first process; determining whether to recognize a second object from the image according to the identified power consumption; A program that causes a computer to execute a process. [Explanation of symbols]

[0073] 10 Image processing device 11 Image acquisition unit 12 Object detection unit 13 Power Management Department 14 Information generation section 15 Network power receiving unit 16 Log Management Department 20 Control equipment 21 Grid setting section 22 Control Department 25 Communications Department C. Imaging device R Transport Robot T1, T2 Obstacles 100 Image processing device 101 CPU 102 ROM 103 RAM 104 Programs 105 Storage device 106 Drive device 107 Communication Interface 108 Input / Output Interface 109 Bus 110 Storage medium 111 Communication Network 121 first processing means 122 Specific means 123 Judgment means 130 Imaging Device

Claims

1. A first processing means for performing a first process of recognizing a first object from images acquired at a predetermined cycle; a determination unit for determining power consumption including the first process; a determination means for determining whether to perform a second process of recognizing a second object from the image in accordance with a comparison between the identified power consumption and a threshold value at a predetermined cycle; An image processing device comprising:

2. 2. The image processing device according to claim 1, the specifying means specifies power consumption including the second process; the determining means determines whether to execute a third process of determining the type of the second object from the image, depending on the power consumption including the specified second process. Image processing device.

3. 3. The image processing device according to claim 2, an estimation means for estimating power consumption including the third processing based on a result of the second processing; the determining means determines whether to execute the third process in accordance with the estimated power consumption. Image processing device.

4. 4. The image processing device according to claim 3, the estimation means estimates the power consumption including the power consumption for the third processing based on the number of the second objects recognized from the image. Image processing device.

5. 5. The image processing device according to claim 2, the determining means changes the timing of performing the second process or the third process depending on the identified power consumption including the power consumption of the first process. Image processing device.

6. 5. The image processing device according to claim 3, the determining means changes the timing of performing the third process in accordance with the estimated power consumption. Image processing device.

7. 7. The image processing device according to claim 1, a detection means for detecting the degree of a malfunction occurring in the image processing device; a setting means for setting a threshold value according to the degree of the defect; the determination means determines whether the second object is to be recognized from the image in accordance with the power consumption and the threshold value. Image processing device.

8. An image processing device comprising: performing a first process of recognizing a first object from images acquired at a predetermined interval; Identifying power consumption including the first process; determining whether to perform a second process of recognizing a second object from the image in accordance with a comparison between the identified power consumption and a threshold value at a predetermined cycle; Image processing methods.

9. 9. The image processing method according to claim 8, The image processing device Identifying power consumption including the second process; determining whether to execute a third process of determining the type of the second object from the image according to the identified power consumption including the power consumption of the second process; Image processing methods.

10. 10. The image processing method according to claim 9, The image processing device estimating power consumption including the third processing based on a result of the second processing; determining whether to execute the third process according to the estimated power consumption; Image processing methods.

11. 11. The image processing method according to claim 10, The image processing device estimating power consumption including the third processing based on the number of the second objects recognized from the image; Image processing methods.

12. 12. An image processing method according to claim 9, further comprising: The image processing device changing a preset timing for performing the second process or the third process according to the identified power consumption including the first process; Image processing methods.

13. 12. The image processing method according to claim 10 or 11, The image processing device changing a preset timing for performing the third process in accordance with the estimated power consumption; Image processing methods.

14. 14. An image processing method according to any one of claims 8 to 13, The image processing device Detect the degree of the malfunction that has occurred in the image processing device, A threshold value is set according to the degree of the defect, determining whether to recognize the second object from the image according to the power consumption and the threshold value; Image processing methods.

15. A first processing means for performing a first process of recognizing a first object from images acquired at a predetermined period; a determination unit for determining power consumption including the first process; a determination means for determining whether to perform a second process of recognizing a second object from the image in accordance with a comparison between the identified power consumption and a threshold value at a predetermined cycle; An information processing system comprising:

16. 16. The information processing system according to claim 15, the specifying means specifies power consumption including the second process; the determining means determines whether to execute a third process of determining the type of the second object from the image, depending on the power consumption including the specified second process. Information processing system.

17. 17. The information processing system according to claim 16, an estimation means for estimating power consumption including the third processing based on a result of the second processing; the determining means determines whether to execute the third process in accordance with the estimated power consumption. Information processing system.

18. 18. The information processing system according to claim 17, the estimation means estimates the power consumption including the power consumption for the third processing based on the number of the second objects recognized from the image. Information processing system.

19. 19. An information processing system according to any one of claims 16 to 18, the determining means changes the timing of performing the second process or the third process depending on the identified power consumption including the power consumption of the first process. Information processing system.

20. 19. The information processing system according to claim 17 or 18, the determining means changes the timing of performing the third process in accordance with the estimated power consumption. Information processing system.

21. 21. An information processing system according to any one of claims 15 to 20, a detection means for detecting the degree of a malfunction occurring in the information processing system; a setting means for setting a threshold value according to the degree of the defect; the determination means determines whether the second object is to be recognized from the image in accordance with the power consumption and the threshold value. Information processing system.

Images