Photographing system, control device and program

Abstract

To reduce a capacity of a memory for storing image data and to improve an image processing speed.SOLUTION: A control device includes travel control means and photographing control means. The travel control means controls travel of a carriage that travels along a photographing plane on which a plurality of subjects to be photographed are arranged at intervals in vertical and horizontal directions. The photographing control means controls a photographing operation of a plurality of cameras mounted in a direction perpendicular to a traveling direction of the carriage in accordance with a photographing timing list set for each of the plurality of cameras.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an imaging system, a control device for this system, and a program for causing a computer to function as the control device. [Background technology]

[0002] For example, there is a solution that uses a camera to capture images of product shelves installed in a store and analyzes the captured images to verify whether the shelf tags attached to the product shelves are displayed correctly. Because such solutions require the recognition of the characters on the shelf tags from the captured images, a so-called narrow-area camera with high resolution and a narrow capture area is suitable. However, many types of products are displayed on the shelves in multiple tiers, and the width and height of the shelves required for display vary depending on the product. Therefore, the shelf tags indicating the prices of each product are not evenly spaced vertically or horizontally on the shelves.

[0003] To avoid this, multiple narrow-area cameras are attached to a cart that moves across the width of the shelf, aligned along the height of the shelf, so that shelf tags can be located anywhere on the shelf. Each narrow-area camera then repeatedly captures images of the shelf as the cart moves. By building such an imaging system, high-resolution images of all shelf tags attached to the shelf can be acquired, and image processing can be performed to recognize the characters on the shelf tags.

[0004] However, in conventional photography systems, each camera repeatedly photographs product shelves regardless of whether shelf labels are present, and therefore images that do not contain shelf labels are also processed, which poses issues that need to be resolved, such as the need for a large memory capacity to store image data and a decrease in image processing speed. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special Publication No. 2019-513274 Summary of the Invention [Problem to be solved by the invention]

[0006] The problem to be solved by the embodiments of the present invention is to provide an imaging system and a control device thereof that can reduce the capacity of the memory that stores image data and improve the image processing speed. [Means for solving the problem]

[0007] In one embodiment, the control device includes a travel control means and a photography control means. The travel control means controls the travel of a dolly that travels along a photography surface on which a plurality of photography targets are arranged at intervals in the vertical and horizontal directions. The photography control means controls the photography operations of a plurality of cameras attached perpendicular to the traveling direction of the dolly in accordance with a photography timing list set for each of the plurality of cameras. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram showing a schematic configuration of an imaging system according to an embodiment. [Figure 2] FIG. 2 is a plan view showing an example of an application scene of the imaging system. [Figure 3] FIG. 3 is a front view showing an example of a product shelf. [Figure 4] FIG. 4 is a schematic diagram showing the data structure of the shelf data table. [Figure 5] FIG. 5 is a schematic diagram showing an image capturing area when the first camera captures an image of a product shelf at a narrow-area image capturing point. [Figure 6] FIG. 6 is an explanatory diagram of the main functions of the processor of the control device in one embodiment. [Figure 7] FIG. 7 is a flowchart showing the main steps of the information processing executed by the processor in accordance with the control program. [Figure 8] FIG. 8 is a flowchart showing a specific procedure for the wide-angle imaging process in FIG. [Figure 9] FIG. 9 is a flowchart showing a specific procedure of the narrow-area imaging process in FIG. [Figure 10] FIG. 10 is a schematic diagram showing an example of a shelf label position image. [Figure 11] FIG. 11 is a schematic diagram showing the image capturing area captured by seven first cameras attached to the image capturing device. [Figure 12] FIG. 12 is a schematic diagram showing the configuration of a timing list. [Figure 13] FIG. 13 is a diagram in which the shelf label position image and the photographed area by the first camera are superimposed. [Figure 14] FIG. 14 is a diagram showing an example of a shooting flag stored in the timing list. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an embodiment of an imaging system and a control device thereof will be described with reference to the drawings.

[0010] In this embodiment, an image capturing system for checking whether the display on the shelf tag attached to the front of the product shelf is correct will be exemplified.

[0011] 1 is a schematic diagram showing the general configuration of an imaging system 100 according to this embodiment. The imaging system 100 includes an imaging device 10 and a control device 20. The imaging device 10 and the control device 20 are connected via, for example, a wireless LAN (Local Area Network). The imaging device 10 and the control device 20 may also be connected via a wired communication cable.

[0012] The photographing device 10 comprises a camera mounting unit 12 mounted on a cart 11. The cart 11 is a vehicle that moves freely on the floor in front of the product shelves to which the shelf tags, which are the photographic targets, are attached. The cart 11 moves autonomously in response to driving instructions from a control device 20. Such a cart 11 can be rephrased as an autonomously moving robot.

[0013] The camera mounting portion 12 is made up of a first camera mounting portion 121 and a second camera mounting portion 122. The first camera mounting portion 121 and the second camera mounting portion 122 are fixed to the upper portion of the dolly 11, respectively.

[0014] The first camera mounting section 121 is a structure in which a plurality of first cameras 13 are mounted in a row at regular intervals in a direction (arrow Z) perpendicular to the traveling direction (arrow X) of the dolly 11, with their lenses facing the same direction. The first cameras 13 are high-resolution narrow-area cameras suitable for photographing shelf labels. In this embodiment, there are seven first cameras 13. The first cameras 13 closest to the dolly 11 are numbered "131," "132," "133," "134," "135," "136," and "137" to distinguish them from the others. In the following description, the reference number "13" will be used to collectively refer to the first cameras, and the reference numbers "131" to "137" will be used to describe them individually.

[0015] The second camera mounting portion 122 is a structure that is approximately the same in shape and size as the first camera mounting portion 121, and has one second camera 14 attached to the center of its height so that its lens faces the same direction as each of the first cameras 13. The second camera 14 is a wide-area camera with a wider angle of view than the first cameras 13. The second camera 14 is a wide-area camera with an angle of view that allows it to capture the entire front of at least one product shelf.

[0016] The control device 20 is a computer device that remotely controls the travel of the dolly 11 in the imaging device 10 and the imaging operations of the first camera 13 and the second camera 14. The control device 20 includes a processor 21, a main memory 22, an auxiliary storage device 23, an imaging device interface 24, a timer 25, a touch panel 26, a communication interface 27, and a system transmission path 28. The system transmission path 28 includes an address bus, a data bus, a control signal line, and the like. The system transmission path 28 connects the processor 21 to each of the other components directly or via a signal input / output circuit, and transmits data signals exchanged between them. The control device 20 constitutes a computer by connecting the processor 21 to the main memory 22, the auxiliary storage device 23, the imaging device interface 24, the timer 25, the touch panel 26, and the communication interface 27 via the system transmission path 28.

[0017] The processor 21 corresponds to the central part of the computer. The processor 21 controls each part to realize various functions of the control device 20 in accordance with an operating system or an application program. The processor 21 is, for example, a CPU (Central Processing Unit).

[0018] The main memory 22 corresponds to the main storage portion of the computer. The main memory 22 includes a nonvolatile memory area and a volatile memory area. The main memory 22 stores an operating system or application programs in the nonvolatile memory area. The main memory 22 may store data required for the processor 21 to execute processes for controlling each part in either the nonvolatile or volatile memory area. The main memory 22 uses the volatile memory area as a work area where data is rewritten by the processor 21 as appropriate. The nonvolatile memory area is, for example, ROM (Read Only Memory). The volatile memory area is, for example, RAM (Random Access Memory).

[0019] The auxiliary storage device 23 corresponds to the auxiliary storage portion of the computer. For example, an EEPROM (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive) can be the auxiliary storage device 23. The auxiliary storage device 23 stores data used by the processor 21 when performing various processes, data created by the processes in the processor 21, etc. The auxiliary storage device 23 may also store the application programs described above.

[0020] The imaging device interface 24 is an interface for wirelessly or wiredly communicating data with the imaging device 10. The imaging device interface 24 transmits data signals relating to driving instructions, such as starting, stopping, driving speed, and direction of the carriage 11, to the imaging device 10. The imaging device interface 24 receives image data captured by the first camera 13 and the second camera 14, respectively.

[0021] The timer 25 is a peripheral circuit that measures a set time in response to an instruction from the processor 21 and notifies the time-out each time the time is up. The set time will be described later.

[0022] The touch panel 26 functions as an input device and a display device for the control device 20. An operator of the control device 20 can operate the touch panel 26 to input data necessary for controlling the imaging device 10 into the control device 20. The operator can also check the image processing results of the image processing device 30, which will be described later, from the information displayed on the touch panel 26.

[0023] The communication interface 27 is an interface for performing data communication with the image processing device 30 connected via a communication line such as the Internet. The image processing device 30 is a computer device having a processing function for checking whether the display on the shelf label is correct from the image of the shelf label photographed by the photographing device 10. Image processing by the image processing device 30 may be realized by cloud computing technology, for example.

[0024] Fig. 2 is a plan view showing an example of an application scene of the photography system 100 according to this embodiment. Fig. 2 illustrates an example in which four product shelves Sa, Sb, Sc, and Sd are installed in a row along the X-axis direction on the floor surface of a store, which corresponds to the XY plane. The carriage 11 of the photography device 10 has a waiting point on the floor surface at a position shown in the figure that is further to the left of the product shelf Sa, which is the leftmost of the four product shelves Sa, Sb, Sc, and Sd in the figure.

[0025] Each of the product shelves Sa, Sb, Sc, and Sd has approximately the same depth and height, but different widths. In one example, the width Wa of the product shelf Sa is equal to the width Wc of the product shelf Sc, but the width Wb of the product shelf Sb is narrower than the width Wa or the width Wc, and the width Wd of the product shelf Sd is even narrower than the width Wb. Each of the product shelves Sa, Sb, Sc, and Sd may have any number of tiers, and the height of each tier may also be arbitrary and may be changed as appropriate depending on the number, size, etc. of the products to be displayed.

[0026] FIG. 3 is a front view of example product shelves Sa and Sb. Product shelves Sc and Sd are not shown. As shown in FIG. 3, product shelf Sa has five shelves. Three shelf tags are attached to each of the bottom first, second, and fourth shelves, and two shelf tags are attached to the third shelf. Four shelf tags are attached to the top fifth shelf. In other words, a total of 15 shelf tags 40 to be photographed are arranged at appropriate intervals in the height direction (i.e., up-down direction) and width direction (i.e., left-right direction) of the product shelf Sa in front of it.

[0027] On the other hand, the product shelf Sb has four tiers. Two shelf tags are attached to the bottom 1st and 3rd tiers, and three shelf tags are attached to the 2nd and 4th tiers. In other words, in front of the product shelf Sb, a total of 10 shelf tags 40 to be photographed are arranged at appropriate intervals in the height direction (i.e., up-down direction) and width direction (i.e., left-right direction) of the product shelf Sb.

[0028] As described above, the product shelf Sa and the product shelf Sb have different numbers of rows. The heights of the rows are also different. Therefore, the shelf tags 40 arranged on the product shelf Sa and the shelf tags 40 arranged on the product shelf Sb are arranged with a vertical offset.

[0029] In this embodiment, the control device 20 stores a shelf data table 50 having the data structure shown in Fig. 4 in, for example, the auxiliary storage device 23. The shelf data table 50 is a data table that describes each shelf data item, such as a shelf ID, X coordinate, Y coordinate, width, and angle, in association with a serial number, as shown in Fig. 4. The shelf data table 50 may be stored in the main memory 22.

[0030] The serial number is a consecutive number starting from "1," and its maximum value corresponds to the number of product shelves to which the shelf tags to be photographed are attached. Therefore, in the example shown in Figure 2, four product shelves Sa, Sb, Sc, and Sd are targeted, and therefore serial numbers from "1" to "4" are entered in the shelf data table 50.

[0031] The shelf ID is a unique code assigned to each product shelf to identify each product shelf Sa, Sb, Sc, and Sd. In the shelf data table 50, the shelf IDs are associated with serial numbers from "1" to "4" and are written in order from the product shelf closest to the waiting point of the trolley 11. Therefore, in the example shown in Fig. 2, the shelf IDs are written in the order of product shelves Sa, Sb, Sc, and Sd for the serial numbers from "1" to "4."

[0032] The X and Y coordinates are coordinate values ​​on the XY plane where the lower end of the trolley 11 closer to the waiting point is located for the product shelves Sa, Sb, Sc, and Sd identified by the corresponding shelf ID. Therefore, in the example shown in Fig. 2, the values ​​of coordinates (Xa, Ya) are described as the X and Y coordinates of the product shelf Sa, the values ​​of coordinates (Xb, Yb) are described as the X and Y coordinates of the product shelf Sb, the values ​​of coordinates (Xc, Yc) are described as the X and Y coordinates of the product shelf Sc, and the values ​​of coordinates (Xd, Yd) are described as the X and Y coordinates of the product shelf Sd.

[0033] The width is the horizontal width of the product shelves Sa, Sb, Sc, and Sd identified by the corresponding shelf ID. Therefore, in the example shown in Figure 2, a value Wa is entered as the width of the product shelf Sa, a value Wb is entered as the width of the product shelf Sb, a value Wc is entered as the width of the product shelf Sc, and a value Wd is entered as the width of the product shelf Sd.

[0034] The angles are the angles formed by the width direction of the product shelves Sa, Sb, Sc, and Sd identified by the corresponding shelf IDs relative to the X-axis of the XY plane. Therefore, in the example shown in Figure 2, 0 degrees is entered as the angles of all of the product shelves Sa, Sb, Sc, and Sd. Incidentally, for example, if the product shelf Sd is installed at a right angle to the width direction of the product shelf Sc, that is, if the product shelves Sc and Sd are installed in an L-shape, 90 degrees is entered as the angle of the product shelf Sd.

[0035] The shelf data table 50 with such a data structure is a data table that describes the data necessary for the processor 21 of the control device 20 to control the movement of the cart 11 of the imaging device 10 and to control the imaging timing of the first camera 13 and the second camera 14.

[0036] In FIG. 2, multiple points indicated by single circles indicate the photographing positions of the first camera 13 relative to the corresponding product shelves Sa, Sb, Sc, and Sd. The first camera 13 is a narrow-area camera used to photograph the shelf labels 40. For this reason, the photographing positions of the first camera 13 are located a distance La away from the front of the product shelves Sa, Sb, Sc, and Sd that is suitable for the first camera 13 to photograph the shelf labels 40. The photographing positions indicated by single circles can also be referred to as narrow-area photographing positions. The narrow-area photographing positions are located at the midpoints of each section obtained by dividing the width of the corresponding product shelves Sa, Sb, Sc, and Sd into regular intervals, at a distance La away from the front of the product shelves Sa, Sb, Sc, and Sd. The regular intervals depend on the photographing area when the first camera 13 photographs the product shelves at the narrow-area photographing positions.

[0037] 5 is a schematic diagram showing a photographing area 60 when the first camera 13 photographs a product shelf at a narrow-area photographing point. The photographing area 60 is a rectangular area with a length T on one side in the left-right direction, i.e., the width direction of the product shelf, and a length H on one side in the up-down direction, i.e., the height direction of the product shelf. The fixed interval is equal to the length T of one side of the photographing area 60 in the left-right direction.

[0038] In FIG. 2, multiple points indicated by double circles indicate the shooting points of the second camera 14 relative to the corresponding product shelves Sa, Sb, Sc, and Sd. The second camera 14, which is a wide-area camera, is used to shoot images of the product shelves Sa, Sb, Sc, and Sd. For this reason, the shooting points of the second camera 14 are located a distance Lb away from the front of the largest product shelf among the product shelves Sa, Sb, Sc, and Sd, which allows the second camera 14 to shoot an image of the entire front of the largest product shelf among the product shelves Sa, Sb, Sc, and Sd. The shooting points indicated by double circles can also be called wide-area shooting points. The wide-area shooting points are points that are a distance Lb away from the front of the product shelves Sa, Sb, Sc, and Sd, and are half the width of the corresponding product shelves Sa, Sb, Sc, and Sd.

[0039] The processor 21 controls the travel of the cart 11 so that the photographing device 10 reaches the photographing position of the first camera 13 or the photographing position of the second camera 14 for each product shelf Sa, Sb, Sc, Sd based on the data in the shelf data table 50. The processor 21 also controls the photographing timing of the first camera 13 and the second camera 14 so that the first camera 13 photographs at the photographing position of the first camera 13 and the second camera photographs at the photographing position of the second camera 14.

[0040] 6 is an explanatory diagram of the main functions of the processor 21 of the control device 20. As shown in the figure, the processor 21 has functions as a travel control means 211, a photography control means 212, a list creation means 213, a photography location determination means 214, an image acquisition means 215, and an output means 216.

[0041] The travel control means 211 has a function of controlling the constant speed travel of the trolley 11 traveling along the front of the product shelves Sa, Sb, Sc, and Sd, i.e., the photographing surface on which multiple photographing objects, i.e., shelf tags 40, are arranged at intervals in the vertical, horizontal and lateral directions.

[0042] The photography control means 212 is a function that controls the photography operation at a narrow-area photography point of multiple cameras, i.e., first cameras 13, mounted perpendicular to the traveling direction of the trolley 11, in accordance with a photography timing list set for each of the multiple first cameras 13.

[0043] The list creation means 213 is a function that identifies the positions of multiple objects to be photographed, i.e., shelf tags 40, placed on the photographing surface, i.e., from an image taken of the front of the product shelves Sa, Sb, Sc, and Sd, and creates a photographing timing list based on the positions of the multiple objects to be photographed and the photographing areas of the multiple first cameras 13.

[0044] The photographing point determination means 214 is a function that determines the photographing point of the photographing surface by the second camera 14, i.e., the wide-area photographing point for each of the product shelves Sa, Sb, Sc, and Sd. Incidentally, the narrow-area photographing point is a point that is moved by half (T / 2) of the length T of one side in the left-right direction of the photographing area 60 from the point of the coordinates (X, Y) of each product shelf Sa, Sb, Sc, and Sd described in the shelf data table 50, and all other points are points that are moved by the length T of one side in the left-right direction of the photographing area 60.

[0045] The image acquisition means 215 is a function that moves the trolley 11 to the wide-area photography point determined by the photography point determination means 214 before controlling the travel of the trolley 11 by the travel control means 211, and performs photography using the second camera 14 to acquire the images necessary for creating a photography timing list.

[0046] The output means 216 has a function of outputting images captured by the plurality of first cameras 13 to the image processing device 30 under the control of the photographing control means 212.

[0047] The functions of the travel control means 211, photography control means 212, list creation means 213, photography location determination means 214, image acquisition means 215, and output means 216 described above are all realized by information processing executed by the processor 21 in accordance with a control program. The control program is a type of application program stored in the main memory 22 or the auxiliary storage device 23. The method for installing the control program in the main memory 22 or the auxiliary storage device 23 is not particularly limited. The registration program can be recorded on a removable recording medium, or the control program can be distributed by communication via a communication network and installed in the main memory 22 or the auxiliary storage device 23. The form of the recording medium is not important as long as it can store a program and is readable by the device, such as a CD-ROM or memory card.

[0048] 7 to 9 are flow charts showing the main steps of the information processing executed by the processor 21 of the control device 20 according to a control program. Also, FIGS. 10 to 13 are diagrams used to provide supplementary explanations of the information processing. Below, the main operations of the imaging system 100 including the control device 20 will be explained using these diagrams.

[0049] For example, when the control start time arrives and the control program is started, the processor 21 starts the procedure shown in the flowchart of Fig. 7. First, in ACT1, the processor 21 reads the shelf data table 50 from the auxiliary storage device 23 or the main memory 22. Then, in ACT2, the processor 21 writes the maximum value of the serial number in the shelf data table 50 into the register memory N as the number of product shelves to be processed. In ACT3, the processor 21 also resets the first counter n.

[0050] After completing the processes in ACT2 and ACT3, the processor 21 counts up the first counter n by "1" in ACT4. Then, the processor 21 checks whether the first counter n has exceeded the value of the register memory N in ACT5.

[0051] If the first counter n does not exceed the value of the register memory N (ACT5, NO), the processor 21 proceeds to ACT6. In ACT6, the processor 21 acquires the nth shelf data (shelf ID, X coordinate, Y coordinate, width, angle) whose serial number is the value of the first counter n from the shelf data table 50. For example, if the first counter n is "1", the processor 21 acquires the shelf data of the product shelf Sa.

[0052] The processor 21 determines a wide-area photography point for the product shelf identified by the shelf ID of the shelf data in ACT 7. For example, when the first counter n is "1", the processor 21 determines the wide-area photography point to be a point that is a distance of 1 / 2 the width Wa from the point of coordinates (Xa, Ya) in a direction tilted at an angle θa with respect to the X axis and that is a distance Lb away from the front of the product shelf Sa.

[0053] The processor 21 moves the dolly 11 of the imaging device 10 to the wide-area imaging point as ACT8. For example, when the first counter n is "1", the dolly 11 is stopped at the waiting point. Then, the processor 21 controls the traveling direction, traveling speed, etc. of the dolly 11 so that the dolly 11 moves from the waiting point to the wide-area imaging point and stops there.

[0054] When the trolley 11 stops at the wide-area photography point, the processor 21 executes wide-area photography processing in ACT9. Details of the wide-area photography processing will be described later. Note that while the wide-area photography processing is being executed, the trolley 11 does not move from the wide-area photography point. When the wide-area photography processing is completed, the processor 21 determines the narrow-area photography start point for the product shelf identified by the shelf ID of the shelf data in ACT10. For example, when the first counter n is "1", the processor 21 determines the point that is a distance La away from the point at coordinates (Xa, Ya) as the narrow-area photography start point.

[0055] In ACT11, the processor 21 moves the carriage 11 of the imaging device 10 to the narrow-area imaging start point. For example, when the first counter n is "1", the processor 21 controls the traveling direction, traveling speed, etc. of the carriage 11 so that the carriage 11 moves from the wide-area imaging point for the product shelf Sa to the narrow-area imaging start point and stops.

[0056] When the trolley 11 stops at the narrow-area photography start point, the processor 21 executes narrow-area photography processing in ACT12. Details of the narrow-area photography processing will be described later. When the narrow-area photography processing is completed, the processor 21 returns to ACT4. The processor 21 further counts up the first counter n by "1". Then, if the first counter n does not exceed the value of the register memory N, the processor 21 executes the processing of ACT6 to ACT12 in the same manner as described above. That is, for example, the processor 21 determines a wide-area photography point for the product shelf Sb based on the shelf data of the product shelf Sb, moves the trolley 11 to the wide-area photography point, and executes the wide-area photography processing. Furthermore, the processor 21 determines a narrow-area photography start point for the product shelf Sb based on the shelf data of the product shelf Sb, moves the trolley 11 to the narrow-area photography start point, and executes the narrow-area photography processing.

[0057] Thereafter, the processor 21 alternately executes wide-angle photography processing and narrow-angle photography processing based on, for example, shelf data of the product shelf Sc and further shelf data of the product shelf Sd. Then, when the first counter n exceeds the value of the register memory N (ACT5, YES), the processor 21 proceeds to ACT13. In ACT13, the processor 21 outputs image data captured by the multiple first cameras 13 to the image processing device 30 via the communication interface 27. In ACT14, the processor 21 also controls the traveling direction, traveling speed, etc. of the cart 11 so that the cart 11 moves to a waiting point and stops there. With this, the processor 21 ends the information processing according to the control program.

[0058] 8 is a flowchart showing the main steps of the wide-angle photography process. The wide-angle photography process for the product shelf unit Sa will be described below. The wide-angle photography process for the other product shelves Sb, Sc, and Sd follows the same procedure, so a description thereof will be omitted here.

[0059] Processor 21, which has entered the wide-area photography process, outputs a photography-on signal as ACT21 to the wide-area camera, that is, second camera 14. Receiving the photography-on signal, second camera 14 performs photography operation. As a result, the second camera 14 photographs the entire front of product shelf Sa, and the image data is sent to control device 20 via photography device interface 24.

[0060] In ACT22, the processor 21 estimates the positions of the shelf tags 40 placed on the product shelf Sa from the image data. To estimate the shelf tag positions, a learning technique for a hierarchical neural network called, for example, a multi-layer neural network (DNN: Deep Neural Network) is used. That is, a DNN model for shelf tag detection is designed, and the control device 20 implements the DNN model. The processor 21 inputs a captured image of the product shelf Sa into the DNN model. Then, through the action of the DNN model, each shelf tag 40 placed on the product shelf Sa is detected, and the position of each is estimated. In ACT23, the processor 21 creates a shelf tag position image 70 in which the positions of the shelf tags 40 are blacked out, as shown in FIG. 10.

[0061] When a product shelf Sb that is narrower in width than the product shelf Sa is photographed by the second camera 14, the shelf tags 40 of the product shelf Sa or the product shelf Sc adjacent to the product shelf Sb may appear in the photographed image. In this case, the processor 21 ignores the shelf tags 40 detected outside the frame of the product shelf Sb and identifies only the positions of the shelf tags 40 detected within the frame.

[0062] The processor 21 calculates the number of narrow-area photographing times Tm as ACT 24. Specifically, the processor 21 divides the width Wa of the product shelf Sa included in the shelf data by the length T of one side in the left-right direction of the photographing area 60 when the first camera 13 photographed the product shelf Sa at the narrow-area photographing point, and sets the quotient (rounded up to the nearest whole number) as the number of narrow-area photographing times Tm.

[0063] 11 is a schematic diagram showing a photographing area 600 photographed by seven first cameras 13 attached to the first camera attachment portion 121 of the photographing device 10. In the figure, photographing area 61 is the photographing area of ​​first camera 131, photographing area 62 is the photographing area of ​​first camera 132, photographing area 63 is the photographing area of ​​first camera 133, photographing area 64 is the photographing area of ​​first camera 134, photographing area 65 is the photographing area of ​​first camera 135, photographing area 66 is the photographing area of ​​first camera 136, and photographing area 67 is the photographing area of ​​first camera 137. In this way, the photographing area 600 is a rectangular range in which the length of one side in the vertical direction, i.e., the height direction of the product shelf Sa, is seven times the length H of one side in the vertical direction of the photographing area 60 and the length of one side in the horizontal direction, i.e., the width direction of the product shelf Sa, is equal to the length T of one side in the horizontal direction of the photographing area 60. Therefore, by sequentially shifting this photographing area 600 in the width direction of the product shelf Sa by the number of narrow-area photographing times Tm at intervals of length T, the entire front of the product shelf Sa comes to be included in the photographing area of ​​the first camera 13.

[0064] The processor 21 creates a timing list 80 (see FIG. 12) as ACT25. FIG. 12 is an example of the timing list 80. The timing list 80 is a data area in a matrix format, with the number of rows corresponding to the number of first cameras 13 and the number of narrow-area shooting times Tm as the number of columns. A shooting flag PQF is written in each data area PQ, which is identified by a row number P and a column number Q. The shooting flag PQF is 1-bit data for identifying whether the first camera 13 corresponding to the row number P will shoot at the narrow-area shooting point corresponding to the column number Q. In this embodiment, the shooting flag PQF indicating that shooting will be performed is set to "1," and the shooting flag PQF indicating that shooting will not be performed is set to "0." Incidentally, at the time of ACT25, all shooting flags PQF are set to either "0" or "1."

[0065] In ACT 26, the processor 21 resets the second counter Q to "0." Next, in ACT 27, the processor 21 counts up the second counter Q by "1." Then, in ACT 28, the processor 21 checks whether the second counter Q has exceeded the number of narrow-angle imaging operations Tm.

[0066] If the second counter Q does not exceed the number of narrow-angle shootings Tm (ACT28, NO), the processor 21 proceeds to ACT29. In ACT29, the processor 21 resets the third counter P to "0." Next, in ACT30, the processor 21 counts up the third counter P by "1." Then, in ACT31, the processor 21 checks whether the third counter P has exceeded the number of first cameras 13.

[0067] If the third counter P does not exceed the number of first cameras 13 (ACT 31, NO), the processor 21 proceeds to ACT 32. In ACT 32, the processor 21 superimposes the photographing area 60 when the first camera 13 corresponding to row number P photographs at the narrow-area photographing point corresponding to column number Q onto the shelf label position image 70.

[0068] In ACT 33, the processor 21 checks whether or not a shelf label 40 is included in the area where the shooting area 60 of the shelf label position image 70 is overlapped. If a shelf label 40 is not included in the area where the shooting area 60 of the shelf label position image 70 is overlapped (ACT 33, NO), the processor 21 proceeds to ACT 34. In ACT 34, the processor 21 sets the shooting flag PQF of the data area PQ specified by the row number P and column number Q to "0." If a shelf label 40 is included in the area where the shooting area 60 of the shelf label position image 70 is overlapped (ACT 33, YES), the processor 21 proceeds to ACT 35. In ACT 35, the processor 21 sets the shooting flag PQF of the data area PQ to "1."

[0069] After completing the processing in ACT 34 or ACT 35, the processor 21 returns to ACT 30. Then, the processor 21 performs the processing from ACT 30 onwards in the same manner as described above. That is, the processor 21 repeatedly executes the processing in ACT 32 to ACT 35 until the third counter P exceeds the number of first cameras 13.

[0070] If the third counter P exceeds the number of first cameras 13 (ACT31, YES), the processor 21 returns to ACT27. Then, the processor 21 performs the processes from ACT27 onwards in the same manner as described above. That is, the processor 21 further counts up the second counter Q by "1". Then, if the second counter Q does not exceed the number of narrow-angle shootings Tm, the processor 21 resets the third counter P to "0". Thereafter, the processor 21 executes the processes of ACT32 to ACT35 each time the third counter P is counted up. Then, if the third counter P exceeds the number of first cameras 13, the processor 21 returns to ACT27 again and further counts up the second counter Q by "1".

[0071] In this way, when the second counter Q exceeds the number of narrow-angle photography times Tm (ACT28, YES), the processor 21 proceeds to ACT36. In ACT36, the processor 21 saves in the auxiliary storage device 23 a timing list 80 in which photography flags PQF of "1" or "0" are written in P*Q data areas PQ corresponding to each row number P and each column number Q. With this, the processor 21 exits the wide-angle photography process for the product shelf Sa.

[0072] 13 is a schematic diagram showing an example in which photographing areas 61, 62, 63, 64, 65, 66, and 67 photographed by seven first cameras 131, 132, 133, 134, 135, 136, and 137 are superimposed on a shelf label position image 70 for a product shelf Sa by shifting them by length T for the number of narrow-area photographs Tm. In this example, the photographing area 61 photographed by the first camera 131 with row number P=1 includes shelf labels in the areas with column numbers Q=2, 4, 5, and 6. As a result, as shown in FIG. 14, photographing flags 12F, 14F, 15F, and 16F are set to "1," and photographing flags 11F and 13F are set to "0."

[0073] Similarly, in the photographing area 62 photographed by the first camera 132 of row number P=2, shelf labels are included in the area of ​​column numbers Q=1, 2, 3, 4, and 5. As a result, as shown in Fig. 14, photographing flags 21F, 22F, 23F, 24F, and 25F become "1," and photographing flag 26F becomes "0."

[0074] The photographing area 63 photographed by the first camera 133 for row number P=3 does not include any shelf labels in the area of ​​all column numbers Q. As a result, as shown in FIG. 14, the photographing flags 31F, 32F, 33F, 34F, 35F, and 36F are all set to "0."

[0075] In the photographing area 64 photographed by the first camera 134 of row number P=4, shelf labels are included in the area of ​​column numbers Q=2, 3, 4, 5. As a result, as shown in Fig. 14, photographing flags 42F, 43F, 44F, and 45F are set to "1," and photographing flags 41F and 46F are set to "0."

[0076] In the photographing area 65 photographed by the first camera 135 of row number P=5, shelf labels are included in the area of ​​column numbers Q=1, 2, 4, 5, 6. As a result, as shown in Fig. 14, photographing flags 51F, 52F, 54F, 55F, and 56F are set to "1," and photographing flag 53F is set to "0."

[0077] The photographing area 66 photographed by the first camera 136 at row number P=6 does not include any shelf labels in the area of ​​all column numbers Q. As a result, the photographing flags 61F, 62F, 63F, 64F, 65F, and 66F are all set to "0."

[0078] The photographing area 67 photographed by the first camera 137 of row number P=7 includes shelf labels in the area of ​​all column numbers Q. As a result, the photographing flags 71F, 72F, 73F, 74F, 75F, and 76F all become "1."

[0079] Thus, in the wide-area photography process for the product shelf Sa, the timing list 80 shown in Fig. 14 is created for the product shelf Sa. Incidentally, it goes without saying that the timing lists 80 created for the product shelf Sb, the product shelf Sc, and the product shelf Sd in the wide-area photography process for the product shelf Sb, the product shelf Sc, and the product shelf Sd respectively have different values ​​for the photography flag PQF.

[0080] 9 is a flowchart showing the main steps of the narrow-angle photography process. The narrow-angle photography process for the product shelf unit Sa will be described below. The narrow-angle photography process for the other product shelves Sb, Sc, and Sd is also performed in the same manner, so a description thereof will be omitted here.

[0081] Having entered the narrow-area imaging process, processor 21 resets fourth counter R to "0" in ACT 41. Next, processor 21 counts up fourth counter R by "1" in ACT 42. Then, processor 21 checks whether fourth counter R is "1" in ACT 43. If fourth counter R is "1" (ACT 43, YES), processor 21 proceeds to ACT 44. Processor 21 sets timeout time t / 2 in timer 25 in ACT 44. Timeout time t / 2 is half the time required for cart 11 to move the length T of one side in the left-right direction of imaging area 60.

[0082] If the fourth counter R is not "1", that is, if it is "2" or greater (ACT 43, NO), the processor 21 proceeds to ACT 45. In ACT 45, the processor 21 checks whether the fourth counter R has exceeded the number of narrow-angle imaging times Tm.

[0083] If the fourth counter R does not exceed the number of narrow-area imaging times Tm (ACT 45, NO), the processor 21 proceeds to ACT 46. In ACT 46, the processor 21 sets a timeout time t in the timer 25. The timeout time t is the time required for the carriage 11 to move the length T of one side of the imaging area 60 in the left-right direction.

[0084] After completing the processing of ACT44 or ACT46, the processor 21 proceeds to ACT47. In ACT47, the processor 21 controls the start of travel of the carriage 11. The processor 21 also starts the timer 25. Then, in ACT48, the processor 21 waits for the timer 25 to time out. When the timer 25 times out (ACT49, YES), the processor 21 proceeds to ACT50. In ACT50, the processor 21 stops the travel of the carriage 11.

[0085] In ACT11 of FIG. 7, in the narrow-area photography process after the trolley 11 has moved to the narrow-area photography start point of the product shelf Sa, the fourth counter R becomes "1", and therefore a timeout period t / 2 is set for the timer 5. In this state, the processor 21 controls the trolley 11 to move at a constant speed along the width direction (X direction) of the product shelf Sa from the narrow-area photography start point at the coordinates (Xa, Ya). The speed of the trolley 11 is the speed at which the trolley 11 moves half the length T of one side in the left-right direction of the photography area 60 while the timer 25 is counting the time t / 2. Therefore, when the timer 25 times out, the trolley 11 stops at a point that is a distance T / 2 away from the narrow-area photography start point at the coordinates (Xa, Ya) in the width direction (X direction) of the product shelf Sa.

[0086] After completing the processing of ACT50, the processor 21 acquires, in ACT51, from the timing list 80, the shooting flag PQF of column R, in which the column number Q matches the value of the fourth counter R. Then, in ACT52, the processor 21 selects the first camera 13, in which the shooting flag PQF is "1."

[0087] Processor 21 outputs a shooting-on signal to the selected first camera 13 as ACT53. It does not output a shooting-on signal to the unselected first camera 13. The first camera 13 that receives the shooting-on signal performs a shooting operation. Processor 21 captures an image captured by the first camera 13 that receives the shooting-on signal as ACT54, and stores it in memory M for storing captured images. Memory M is part of the volatile area in main memory 22.

[0088] Therefore, when the fourth counter R is "1", the processor 21 acquires the shooting flags 11F, 21F, 31F, 41F, 51F, 61F, and 71F. As shown in FIG. 14, the shooting flag 11F is "0", the shooting flag 21F is "1", the shooting flag 31F is "0", the shooting flag 41F is "0", the shooting flag 51F is "1", the shooting flag 61F is "0", and the shooting flag 71F is "1". Therefore, the processor 21 outputs a shooting-on signal to the first cameras 132, 135, and 137. The processor 21 does not output a shooting-on signal to the first cameras 131, 133, 134, and 136. As a result, the images captured by the first cameras 132, 135, and 137 are stored in the memory M.

[0089] When all of the images captured by the first camera 132, the first camera 135, and the first camera 137 have been stored in the memory M, the processor 21 returns to ACT42. That is, the processor 21 further counts up the fourth counter R by "1." Therefore, the fourth counter R becomes "2," which does not exceed the number of narrow-angle imaging operations Tm, so the processor 21 sets the timer 25 to a timeout time t. The processor 21 then controls the start of travel of the dolly 11 and starts the timer 25. When the timer 25 times out, the processor 21 stops the travel of the dolly 11. The speed of the dolly 11 at this time is also the speed at which the dolly 11 moves half the length T of one side in the left-right direction of the imaging area 60 while the timer 25 counts time t / 2. In other words, the speed at which the dolly 11 moves the length T of one side in the left-right direction of the imaging area 60 while the timer 25 counts time t. Therefore, the dolly 11 stops at a point that is a distance of 3T / 2 in the width direction (X direction) of the product shelf Sa from the narrow-area photography start point at coordinates (Xa, Ya).

[0090] At this time, processor 21 acquires shooting flags 12F, 22F, 32F, 42F, 52F, 62F, and 72F. As shown in FIG. 14, shooting flag 12F is "1," shooting flag 22F is "1," shooting flag 32F is "0," shooting flag 42F is "1," shooting flag 52F is "1," shooting flag 62F is "0," and shooting flag 72F is "1." Therefore, processor 21 outputs a shooting-on signal to first camera 131, first camera 132, first camera 134, first camera 135, and first camera 137. It does not output a shooting-on signal to first camera 133 and first camera 136. As a result, images captured by first camera 131, first camera 132, first camera 134, first camera 135, and first camera 137 are stored in memory M.

[0091] Next, the fourth counter R becomes "3." In this case, the fourth counter R does not exceed the number of narrow-area photographs Tm, so the processor 21 sets the timeout time t in the timer 25. The processor 21 then controls the start of travel of the cart 11 and starts the timer 25. When the timer 25 times out, the processor 21 stops the travel of the cart 11. The speed of the cart 11 at this time is also the same as when the fourth counter R is "1" or "2." Therefore, the cart 11 stops at a point that is a distance of 5T / 2 in the width direction (X direction) of the product shelf Sa from the narrow-area photographing start point at the coordinates (Xa, Ya).

[0092] At this time, processor 21 acquires shooting flags 13F, 23F, 33F, 43F, 53F, 63F, and 73F. As shown in FIG. 14, shooting flag 13F is "0," shooting flag 23F is "1," shooting flag 33F is "0," shooting flag 43F is "1," shooting flag 53F is "0," shooting flag 63F is "0," and shooting flag 73F is "1." Therefore, processor 21 outputs a shooting-on signal to first camera 132, first camera 134, and first camera 137. It does not output a shooting-on signal to first camera 131, first camera 133, first camera 135, and first camera 136. As a result, images captured by first camera 132, first camera 134, and first camera 137 are stored in memory M.

[0093] Thereafter, each time the fourth counter R counts up to "4", "5", or "6", the processor 21 executes the same processing as when the fourth counter R is "2" or "3". As a result, when the fourth counter R is "4", the images taken by the first camera 131, the first camera 132, the first camera 134, the first camera 135, and the first camera 137 are stored in the memory M. When the fourth counter R is "5", the images taken by the first camera 131, the first camera 132, the first camera 134, the first camera 135, and the first camera 137 are also stored in the memory M. When the fourth counter R is "6", the images taken by the first camera 131, the first camera 135, and the first camera 137 are stored in the memory M.

[0094] Thereafter, the fourth counter R becomes "7" and exceeds the number of narrow-area photographing operations Tm. When the fourth counter R exceeds the number of narrow-area photographing operations Tm (ACT45, YES), the processor 21 proceeds to ACT53. In ACT53, the processor 21 clears the timing list 80. With this, the processor 21 exits the narrow-area photographing process for the product shelf Sa.

[0095] Thereafter, the processor 21 determines a wide-area photography point for the product shelf Sb, moves the cart 11 to the wide-area photography point, and executes wide-area photography processing for the product shelf Sb. Next, the processor 21 determines a narrow-area photography start point for the product shelf Sb, moves the cart 11 to the narrow-area photography start point, and executes narrow-area photography processing for the product shelf Sb.

[0096] Here, the processor 21 performs the processing of ACT47 to ACT50 in Fig. 9 to realize the function of the travel control means 211. The processor 21 performs the processing of ACT51 to ACT53 in Fig. 9 to realize the function of the shooting control means 212.

[0097] The processor 21 performs the processing of ACT6 and ACT7 in Fig. 8 to realize the function of a photographing location determination means 214. The processor 21 performs the processing of ACT8 in Fig. 8 and ACT21 in Fig. 9 to realize the function of an image acquisition means 215. The processor 21 performs the processing of ACT22 and ACT36 in Fig. 9 to realize the function of a list creation means 213. The processor 21 performs the processing of ACT13 in Fig. 8 to realize the function of an output means 216.

[0098] In this way, the control device 20 having the functions of the travel control means 211 and the photography control means 212 prevents each first camera 13 from capturing an image that does not include a shelf tag 40. This makes it possible to save the capacity of the memory M that stores the image data captured by each first camera 13. Furthermore, because the control device 20 does not process image data that does not include a shelf tag 40, it is possible to increase the overall speed required for data processing.

[0099] The control device 20 also has a list creation means 213. Therefore, a separate device for creating the timing list 80 is not required, and the effort required to create the timing list 80 can be reduced.

[0100] Furthermore, the control device 20 has a photographing location determination means 214 and an image acquisition means 215. The control device 20 creates a timing list 80 using the list creation means 213 based on the images acquired by the image acquisition means 215. Therefore, for example, even if the attachment position of a shelf tag 40 attached to a product shelf is changed, the timing list 80 after the change can be easily created, thereby improving versatility.

[0101] Furthermore, the control device 20 has output means 216. Therefore, the image processing device 30 does not process unnecessary image data in which the shelf label 40 is not photographed, which has the effect of reducing the load on the image processing device 30. In addition, the amount of communication traffic between the control device 20 and the image processing device 30 can also be reduced.

[0102] Thus, it is possible to provide an imaging system 100 that can reduce the memory capacity for storing image data and improve the image processing speed.

[0103] Although the embodiment of the imaging system 100 and the control device 20 thereof has been described above, the embodiment is not limited to this. For example, the control device 20 can be applied to applications other than recognizing characters on shelf tags 40 placed on product shelves. For example, the control device 20 can also be applied to an imaging system that uses a first camera to capture an image of a barcode printed on a cardboard box placed randomly in a warehouse or the like, and recognizes the barcode from the captured image.

[0104] In the above embodiment, the height of each product shelf is the same. However, the height of each product shelf may be different. In this case, the number of first cameras 13 to be attached to the first camera attachment portion 121 may be determined according to the height of the highest product shelf.

[0105] In the above embodiment, the fixed interval between adjacent narrow-area photographing points is set to be equal to the length T of one side in the left-right direction of the photographing area 60. This interval may be set to be slightly narrower than the length T, so that the images photographed sequentially by the same first camera 13 partially overlap.

[0106] In the above embodiment, an example was given of a case where wide-angle photography processing is performed on one product shelf to create a timing list 80, and then narrow-angle photography processing is performed using that timing list 80, and then that timing list 80 is cleared. In this regard, it is also possible to first perform wide-angle photography processing on two or more product shelves in sequence to create respective timing lists 80, and then perform narrow-angle photography processing for each product shelf using the timing list 80 for that product shelf.

[0107] Furthermore, the timing list 80 does not necessarily have to correspond to one product shelf. If two product shelves Sa, Sb are adjacent to each other as shown in Fig. 3 and the second camera 14 can capture the entire area in front of both product shelves Sa, Sb, one timing list 80 may be created for the two product shelves Sa, Sb. This reduces the number of movements of the cart 11 required for wide-area photographing processing, thereby improving processing efficiency.

[0108] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope of the invention and the scope of the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0109] 10...photography device, 11...cart, 12...camera mounting section, 13...first camera, 14...second camera, 20...control device, 21...processor, 22...main memory, 23...auxiliary storage device, 24...photography device interface, 25...timer, 26...touch panel, 27...communication interface, 28...system transmission path, 30...image processing device, 40...shelf label, 50...shelf data table, 60...photography area, 70...shelf label position image, 121...first camera mounting section, 122...second camera mounting section, 211...travel control means, 212...photography control means, 213...list creation means, 214...photography location determination means, 215...image acquisition means, 216...output means.

Claims

1. a travel control means for controlling the travel of a carriage that travels along an imaging plane on which a plurality of imaging targets are arranged at intervals in the vertical and horizontal directions; a photographing control means for controlling the photographing operations of a plurality of cameras attached in a direction perpendicular to the traveling direction of the carriage in accordance with a photographing timing list set for each of the plurality of cameras; A control device comprising:

2. a list creation means for identifying positions of the plurality of subjects arranged on the photographing surface from an image photographed on the photographing surface, and creating the photographing timing list based on the positions of the plurality of subjects and the photographing areas of the plurality of cameras; The control device of claim 1 further comprising:

3. the dolly includes a second camera for capturing an image of the imaging surface, separate from the plurality of cameras; a photographing point determination means for determining a photographing point of the photographing surface by the second camera; an image acquisition means for moving the dolly to the photographing point before controlling the travel of the dolly by the travel control means, and causing the second camera to perform a photographing operation to acquire images necessary for creating the photographing timing list; The control device of claim 2 further comprising:

4. an output means for outputting images captured by the plurality of cameras to an image processing device under the control of the photography control means; 4. The control device according to claim 1, further comprising:

5. A control device according to any one of claims 1 to 4; a cart that travels along a photographing surface on which a plurality of photographing targets are arranged at intervals in the vertical and horizontal directions, and that has a plurality of cameras attached in a direction perpendicular to the traveling direction; An imaging system comprising:

6. The control device computer, a travel control means for controlling the travel of a carriage that travels along an imaging plane on which a plurality of imaging targets are arranged at intervals in the vertical and horizontal directions; and an imaging control means for controlling the imaging operations of a plurality of cameras attached in a direction perpendicular to the traveling direction of the carriage in accordance with an imaging timing list set for each of the plurality of cameras; A program to function as a

Images