Information processing system

Abstract

To provide an information processing system to be used for identification / price reading of commodities displayed on a display shelf.SOLUTION: An information processing system 1 for image information for photographing a display shelf, includes: a photographing processing part 20 for detecting a movement amount to photograph the display shelf; a correction processing part 22 for performing correction processing to image information photographed by the photographing processing part; an object detection processing part 23 for detecting an object reflected in the image information subjected to the correction processing; a cut-off determination processing part 24 for making a cut-off determination by using the presence / absence of the object detected from the image information or a positional relation of the detected object, and performing a prescribed warning in the case of determining that cut-off occurs; an image selection processing part 25 for selecting image information to be an object of commodity identification processing or price reading processing from a plurality of pieces of image information photographed by the photographing processing part or the image information after the correction processing by the correction processing part; and a recognition processing part for executing the commodity identification processing or the price reading processing to the selected image information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing system used to identify and read the prices of products displayed on a display shelf. [Background technology]

[0002] In various stores such as convenience stores and supermarkets, it is common for products to be sold by placing them on display shelves. Therefore, by displaying multiple products on a display shelf, even if one product is purchased, another person can purchase the same product. Also, products that are displayed in large quantities in prominent locations attract attention and sell more than other products, resulting in a competitive advantage / inferiority relationship. Therefore, it is important in product sales strategies to manage where and how many products are displayed on the display shelves, as well as the prices of those products.

[0003] Therefore, by photographing the display shelves with a photographing device such as a camera and automatically recognizing the objects in the image information, it is possible to ascertain the display positions and numbers of products, the displayed products, and the prices written on the price tags (price cards). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 5-334409 [Patent Document 2] Japanese Patent Application Publication No. 5-342230 Summary of the Invention [Problem to be solved by the invention]

[0005] When photographing products displayed on display shelves with an imaging device, it is common that various imaging conditions, such as the space of the imaging location, the presence of obstacles, etc. For example, if the width of the aisle between the display shelves where the products are displayed is narrow, it is not possible to photograph the display shelves with a single image, and one display shelf must be photographed with multiple images.

[0006] When identifying products displayed on a display shelf, processing is performed on a shelf-by-shelf basis, so a single display shelf is constructed using multiple pieces of image information, and product identification processing and price reading processing are performed based on this.

[0007] When photographing a display shelf in sections, combining multiple pieces of image information into a single piece of image information will result in image distortion (steps and blurring) at the combined points, which will affect the product identification and price identification processes. For this reason, not combining images is better for the accuracy of the product identification and price reading processes, but the photographer is required to take as few images as possible, which increases the burden of the photography work. For this reason, it is preferable for the photographer to take the photos mechanically, but in that case, the number of photos may increase. In that case, there is a problem that the image information to be processed increases, leading to an overall increase in processing. Therefore, there is a need to reduce the number of image information to be processed while reducing the burden of photography.

[0008] Furthermore, when the display shelf is divided and photographed, there may be gaps between image information, resulting in areas that cannot be photographed. Or the left and right sides of the display shelf may be cut off. In such cases, it is necessary to photograph again, but these situations become clear at the stage of actually performing image analysis. Therefore, after the stage of performing image analysis, it becomes necessary to photograph again. However, in order to photograph, it is necessary to go to the store again and photograph, which significantly reduces work efficiency. [Means for solving the problem]

[0009] In view of the above problems, the present inventor has invented an information processing system for identifying products displayed on a display shelf.

[0010] The first invention is an information processing system for image information photographed of a display shelf, the information processing system having: a photography processing unit that detects the amount of movement and photographs the display shelf; a correction processing unit that performs correction processing on the image information photographed by the photography processing unit; an object detection processing unit that detects objects appearing in the image information after the correction processing; an out-of-view determination processing unit that performs an out-of-view determination using the presence or absence of an object and / or the positional relationship of the detected object detected from the image information, and issues a prescribed warning if it determines that the object is out of view; an image selection processing unit that selects image information to be subjected to product identification processing and / or price reading processing from multiple pieces of image information photographed by the photography processing unit or image information after correction processing by the correction processing unit; and a recognition processing unit that executes the product identification processing and / or the price reading processing on the selected image information.

[0011] By using the information processing system of the present invention, the person in charge of photography can mechanically photograph the display shelves according to the information from the information processing system of the present invention, and the image information to be processed is appropriately selected, thereby reducing the burden of photography and the number of image information to be processed.

[0012] Furthermore, as described above, if the captured image information contains cut-off image information, a warning can be issued at the site where the image is being captured, thereby preventing a decrease in work efficiency, such as having to go back to the store to capture the image again.

[0013] The second invention is an information processing system for image information photographed of a display shelf, the information processing system having an imaging processing unit that detects the amount of movement and photographs the display shelf, a correction processing unit that performs correction processing on the image information photographed by the imaging processing unit, an object detection processing unit that detects objects appearing in the image information after the correction processing, an image selection processing unit that selects image information to be subjected to product identification processing and / or price reading processing from multiple pieces of image information photographed by the imaging processing unit or image information after correction processing by the correction processing unit, and a recognition processing unit that executes the product identification processing and / or the price reading processing on the selected image information.

[0014] By using the information processing system of the present invention, the person in charge of photography can mechanically photograph the display shelves while appropriately selecting the image information to be processed, thereby reducing the burden of photography while also reducing the number of image information to be processed.

[0015] In the above-mentioned invention, the image selection processing unit can be configured as an information processing system in which any object in the image information to be processed is treated as a tracking object, and non-adjacent image information containing the tracking object is selected as image information to be skipped to, and the image information to be subjected to the product identification processing and / or the price reading processing is selected.

[0016] In the above-mentioned invention, the image selection processing unit can be configured as an information processing system in which an object near the end of the image information to be processed is treated as a tracking object, and image information that includes the tracking object and is two or more positions away from the image information to be processed is treated as skipped image information, and selected as image information to be subjected to the product identification processing and / or the price reading processing.

[0017] As in these inventions, by determining image information that includes a tracking object in image information to be processed, it is possible to select image information to be skipped to. This makes it possible to exclude the area between the image information to be processed and the image information to be skipped from the processing target, thereby reducing the number of image information to be processed.

[0018] In the above-mentioned invention, the image selection processing unit can be configured as an information processing system that uses position information of a tracking object in the image information to be processed and the amount of movement to determine whether the tracking object in the image information to be processed is included in the shooting range of the image information preceding the image information to be processed.

[0019] There are various processes for determining whether a tracking object is included in image information, but by using the process of the present invention, the determination can be made without matching image information of the tracking object's area, thereby reducing the processing load.

[0020] In the above-described invention, the information processing system can be configured as an information processing system having an out-of-view determination processing unit that performs out-of-view determination using the presence or absence of an object detected from the image information and / or the positional relationship of the detected object.

[0021] With the configuration of the present invention, it is possible to determine whether the captured image information includes cut-off image information at the store where the image is captured.

[0022] In the above-mentioned invention, the cut-off determination processing section may be configured as an information processing system that, when it determines that there is cut-off image information, displays a predetermined warning.

[0023] With the configuration of the present invention, if the captured image information contains cut-off image information, a warning can be issued at the site where the image is being captured, thereby preventing a decrease in work efficiency, such as having to go back to the store to capture the image again.

[0024] The eighth invention is an information processing system for image information photographed of a display shelf, the information processing system having an imaging processing unit that photographs the display shelf, a correction processing unit that performs correction processing on the image information photographed by the imaging processing unit, an object detection processing unit that detects objects appearing in the image information after the correction processing, an image cut-off determination processing unit that performs an image cut-off determination using the presence or absence of an object and / or the positional relationship of the detected object detected from the image information, and issues a prescribed warning if it determines that an object is cut off, and a recognition processing unit that executes the product identification processing and / or the price reading processing on the image information.

[0025] With the configuration of the present invention, if the captured image information contains cut-off image information, a warning can be issued at the site where the image is being captured, thereby preventing a decrease in work efficiency, such as having to go back to the store to capture the image again.

[0026] In the above-mentioned invention, the cut-off determination processing unit can be configured as an information processing system that determines cut-off image information in the vertical direction based on the presence or absence of space above the shelf or stock product area, and area below the shelf, from the multiple captured image information.

[0027] In the above-mentioned invention, the cut-off determination processing unit can be configured as an information processing system that determines cut-off image information in the left-right direction by detecting whether, among the multiple captured image information, an area where a product is placed or a price display area is detected in the left-right center at a size equal to or larger than a predetermined size, and whether left and right cuts or left and right structural materials are within a predetermined range from the left and right edges of the image information.

[0028] The cut-off determination process can be carried out as in these inventions.

[0029] In the above-mentioned invention, the photographing processing section may be configured as an information processing system that automatically performs photographing when it detects that the distance on the angle of view has moved by a predetermined value or more.

[0030] With the configuration of the present invention, the person in charge of photography can automatically take pictures simply by mechanically changing the orientation of the photography device in a fixed direction, thereby reducing the burden on the person in charge of photography.

[0031] A twelfth invention is an information processing system for image information photographed of a display shelf, comprising an image photographing processing unit that photographs the display shelf, and a recognition processing unit that executes the product identification processing and / or the price reading processing on the image information, wherein the image photographing processing unit performs correction processing on the photographed image information until a predetermined image photographing end condition is satisfied, and when it detects that a reference point in the photographed image information or the corrected image information has moved to a predetermined location due to movement of a portable communication terminal, the information processing system performs photographing or accepts a photographing operation.

[0032] With the configuration of the present invention, when taking pictures with a portable communication terminal, the number of images taken can be reduced because the image is taken when the reference point moves to a specified location or the image is taken by receiving a shooting operation, which reduces the number of times correction processing, object detection processing, etc. are performed, thereby reducing the processing load.

[0033] In the above-mentioned invention, the shooting processing unit can be configured as an information processing system that performs correction processing on the captured image information until a shooting end condition is satisfied using an object captured in the image information, detects an object captured in the corrected image information, sets a predetermined location at or near an object in the direction of movement of the portable communication terminal among the objects captured in the image information, and performs shooting or accepts a shooting operation when it detects that a reference point in the captured image information or the corrected image information has moved to the predetermined location due to the movement of the portable communication terminal.

[0034] As in the present invention, a marker may be set on or near an object detected from captured image information, and processing may be performed.

[0035] In the above-mentioned invention, the shooting processing unit can be configured as an information processing system that calculates the amount of movement from the reference point to the specified location and displays movement according to the amount of movement and / or the direction of movement.

[0036] By carrying out the processing of the present invention, the person in charge of photography can easily recognize how much and in what direction the portable communication terminal should be moved.

[0037] In the above-mentioned invention, the photographing processing section may be configured as an information processing system that issues a predetermined notification when it detects that the reference point has moved to the predetermined location.

[0038] As with the present invention, when it is detected that the portable communication terminal has moved to a specified location, a message is displayed indicating not to move the portable communication terminal, and the person in charge of photography can operate the portable communication terminal in accordance with the instructions.

[0039] In the above-mentioned invention, the correction processing unit has a sensor information input reception processing unit that receives input of pitch angle and roll angle information of the imaging device or information processing device and distance information to an arbitrary point on an object to be photographed, and an image transformation processing unit that performs image transformation processing on the image information, and the image transformation processing unit can be configured as an information processing system that uses the distance information and the pitch angle and roll angle information to calculate a regression line of a point cloud including a plurality of the points, estimates yaw angle information of the imaging device or information processing device, and transforms the image information to a position directly facing the imaging device using the estimated yaw angle information.

[0040] The configuration of the present invention enables correction processing that does not impose a large processing load, and therefore correction processing can be performed even on a portable communication terminal such as a smartphone that does not have high capabilities.

[0041] In the above-mentioned invention, the image transformation processing unit can be configured as an information processing system that converts distance information to an arbitrary point on the object to be photographed into three-dimensional information, rotates the arbitrary point based on pitch angle and roll angle information, and calculates a regression line of a point cloud including the arbitrary point when projected onto a horizontal plane, thereby estimating the yaw angle information.

[0042] It is preferable to carry out the correction process as in the present invention.

[0043] In the above-mentioned invention, the correction processing section may be configured as an information processing system having an output processing section that outputs correction parameters for performing image transformation processing on the image information.

[0044] Since the product identification process / price reading process has a high processing load, it is preferable to perform the process on a server with high processing power, rather than on a portable communication terminal with low processing power, such as a smartphone. In this case, the image information to be the subject of the product identification process / price reading process is uploaded from the portable communication terminal to the server, but when uploading the corrected image information, blank data is included, increasing the amount of data. Therefore, by outputting correction parameters as in the present invention, it becomes possible to upload the image information before correction and the correction parameters, thereby reducing the amount of data to be uploaded.

[0045] In the above-mentioned invention, the recognition processing unit can be configured as an information processing system that performs the product identification process on a product area detected by the object detection processing unit, and / or performs the price reading process on a price display area detected by the object detection processing unit.

[0046] As in the present invention, rather than performing product identification processing / price display area processing on the entire image information, by performing product identification processing / price display area processing on product areas / price display areas detected by object detection processing, it is possible to reduce the processing load and speed up processing.

[0047] In the above-mentioned invention, the information processing system has an upload processing unit that uploads part or all of the pre-correction image information and correction parameters for correction for the image information selected by the image selection processing unit, and a correction reproduction processing unit that reproduces the correction using part or all of the uploaded pre-correction image information and the correction parameters, and the recognition processing unit can be configured as an information processing system that performs product identification processing and / or price reading processing on part or all of the corrected image information reproduced by the correction reproduction processing unit.

[0048] Since the product identification process / price reading process has a high processing load, it is preferable to perform the process on a server with high processing power, rather than on a portable communication terminal with low processing power, such as a smartphone. In this case, the image information to be the subject of the product identification process / price reading process is uploaded from the portable communication terminal to the server, but when uploading the corrected image information, blank data is included, increasing the amount of data. Therefore, by outputting correction parameters as in the present invention, it becomes possible to upload the image information before correction and the correction parameters, thereby reducing the amount of data to be uploaded.

[0049] The information processing system of the first invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes a computer to function as an image capture processing unit that detects the amount of movement and captures an image of a display shelf, a correction processing unit that performs correction processing on image information captured by the image capture processing unit, an object detection processing unit that detects an object appearing in the image information after the correction processing, an out-of-view determination processing unit that performs out-of-view determination using the presence or absence of an object detected from the image information and / or the positional relationship of the detected object and issues a predetermined warning if it is determined that the object is out-of-view, an image selection processing unit that selects image information to be subjected to product identification processing and / or price reading processing from multiple pieces of image information captured by the image capture processing unit or image information after correction processing by the correction processing unit, and a recognition processing unit that executes the product identification processing and / or the price reading processing on the selected image information.

[0050] The information processing system of the second invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes a computer to function as an image capture processing unit that detects the amount of movement and captures an image of a display shelf, a correction processing unit that performs correction processing on image information captured by the image capture processing unit, an object detection processing unit that detects an object appearing in the image information that has been corrected, an image selection processing unit that selects image information to be subjected to a product identification process and / or a price reading process from a plurality of pieces of image information captured by the image capture processing unit or image information after correction processing by the correction processing unit, and a recognition processing unit that executes the product identification process and / or the price reading process on the selected image information.

[0051] The information processing system of the eighth invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes a computer to function as an image capture processing unit that captures an image of a display shelf, a correction processing unit that performs a correction process on image information captured by the image capture processing unit, an object detection processing unit that detects an object appearing in the image information after the correction process, an out-of-view determination processing unit that performs an out-of-view determination process using the presence or absence of an object detected from the image information and / or the positional relationship of the detected object and issues a prescribed warning when it is determined that an object is out-of-view, and a recognition processing unit that performs the product identification process and / or the price reading process on the image information.

[0052] The information processing system of the twelfth invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes a computer to function as an image capturing processing unit that captures an image of a display shelf and a recognition processing unit that executes the product identification processing and / or the price reading processing on the image information, in which the image capturing processing unit performs a correction processing on the captured image information until a predetermined image capturing end condition is satisfied, and performs an image capturing operation or accepts an image capturing operation when it detects that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to the movement of a portable communication terminal.

[0053] The portable communication terminal used in the information processing system of the present invention can be realized by loading and executing the information processing program of the present invention into a portable communication terminal. That is, the information processing program causes a portable communication terminal used in an information processing system that photographs a display shelf and recognizes products and / or prices displayed in image information to function as an image processing unit that detects the amount of movement by the image capture device of the portable communication terminal and photographs the display shelf, a correction processing unit that performs correction processing on the image information photographed by the image capture processing unit, an object detection processing unit that detects objects appearing in the image information that has been corrected, an image selection processing unit that selects image information to be subjected to product identification processing and / or price reading processing from multiple pieces of image information photographed by the image capture processing unit or image information corrected by the correction processing unit, and an upload processing unit that uploads part or all of the selected image information in order to execute product identification processing and / or price reading processing using the image information selected by the image selection processing unit.

[0054] The portable communication terminal used in the information processing system of the present invention can be realized by loading and executing the information processing program of the present invention into a portable communication terminal. That is, the information processing program is characterized in that it causes a portable communication terminal used in an information processing system that photographs a display shelf and recognizes products and / or prices displayed in the image information to function as an image processing unit that photographs the display shelf and an upload processing unit that uploads a part or all of the photographed image information or corrected image information in order to execute a product identification process and / or a price reading process using the image information, and the image processing unit performs a correction process on the photographed image information until a predetermined photographing end condition is satisfied, and when it detects that a reference point in the photographed image information or the corrected image information has moved to a predetermined position due to the movement of the portable communication terminal, performs photographing or accepts a photographing operation. Effect of the Invention

[0055] By using the information processing system of the present invention, it is possible to reduce the number of image information to be processed while reducing the burden of photography. In addition, since it is possible to determine image information that is out of date at the store where the photography is performed, it is possible to retake the photograph on the spot, thereby preventing a decrease in work efficiency. [Brief description of the drawings]

[0056] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Diagram 3] 4 is a flowchart showing an example of the entire process in the information processing system of the present invention. [Figure 4] 5 is a flowchart showing an example of a photographing process in the information processing system of the present invention. [Diagram 5]10 is a flowchart showing an example of an image selection process in the information processing system of the present invention. [Figure 6] 13 is a flowchart showing an example of a shelf structure analysis process in the information processing system of the present invention. [Figure 7] 10 is a flowchart showing an example of a skip process in the information processing system of the present invention. [Figure 8] FIG. 13 is a diagram showing an example of object labels and their meanings. [Figure 9] FIG. 13 is a diagram showing an example of a process for determining whether or not an image is cut off in the vertical direction, and is a diagram showing an example of a case where a SpaceOverRack area is detected as the space above a shelf. [Figure 10] FIG. 13 is a diagram showing an example of a process for determining whether or not an image is cut off in the vertical direction, and is a diagram showing an example of a case where a Stock area is detected as a stock commodity area. [Figure 11] FIG. 13 is a diagram showing an example of a process for determining whether or not the image is cut off in the vertical direction, and shows an example in which a RackBottom area is detected as the space under a shelf. [Figure 12] FIG. 13 is a diagram showing a schematic diagram of a case where one display shelf is continuously photographed using a plurality of pieces of image information. [Figure 13] FIG. 10 is a diagram illustrating an image selection process. [Figure 14] FIG. 13 is a diagram showing a schematic diagram of another example in which one display shelf is continuously photographed using a plurality of pieces of image information. [Figure 15] 10A and 10B are diagrams illustrating an example of image information before correction and image information after correction processing in a correction processing unit. [Figure 16] FIG. 13 is a diagram showing an example of a shelf structure table for image information #0. [Figure 17] 13 is an example of a table showing rules for selecting an action based on an object relationship. [Figure 18] FIG. 13 is a diagram showing an example of a shelf structure table for image information #5. [Figure 19] FIG. 13 is a diagram showing an example of a shelf structure table for image information #11. [Figure 20]FIG. 13 is a diagram showing an example of a shelf structure table for image information #14. [Figure 21] FIG. 11 is a block diagram illustrating an example of the configuration of a correction processing unit according to a second embodiment. [Figure 22] 13 is a flowchart illustrating an example of a correction process of a correction processing unit according to a second embodiment. [Figure 23] 13 is an example of image information (image information before correction) to be input. [Figure 24] FIG. 2 is a diagram illustrating a relationship between a target surface and a three-dimensional space. [Diagram 25] 13 is an example of image information (corrected image information) to be output. [Figure 26] FIG. 13 is a front view of the rotated point cloud mapped onto a vertical surface. [Figure 27] FIG. 13 is a top view of the rotated point cloud mapped onto a horizontal plane. [Figure 28] 11 is a diagram illustrating a process in an angle estimation processing unit. FIG. [Figure 29] 11A and 11B are diagrams illustrating processes in an image transformation processing unit. [Diagram 30] 11A and 11B are diagrams illustrating deviations in distance information when an object with many projections and recesses is photographed. [Diagram 31] FIG. 13 is a diagram illustrating a method of randomly selecting a group of points from a matrix, among the group of points on which distance information is plotted. [Diagram 32] 10A and 10B are diagrams showing image information and resulting abnormal values ​​of distance information when unevenness is captured on the left and right sides of an intended subject to be photographed; [Diagram 33] FIG. 11 is a diagram illustrating an example of a modified correction process in the second embodiment. [Diagram 34] FIG. 13 is a diagram illustrating an example of a flowchart of a skip process in the third embodiment. [Diagram 35] FIG. 13 is a diagram illustrating a skip process in the third embodiment. [Diagram 36] FIG. 23 is a block diagram illustrating an example of a configuration of an information processing system according to a sixth embodiment. [Figure 37]FIG. 23 is an example of a conceptual diagram that illustrates the process of the sixth embodiment. [Figure 38] 23 is a flowchart showing an example of the entire process in the sixth embodiment. [Figure 39] 23 is a flowchart showing an example of an imaging process in the sixth embodiment. [Diagram 40] 11A and 11B are diagrams showing an example of image information before and after correction when photographing is performed for the first time. [Diagram 41] FIG. 41 is a diagram showing an example of a state in which an object detection process is performed on the image information of FIG. 40(b). [Diagram 42] FIG. 13 is a diagram showing an example of a state in which a moving display is superimposed. [Diagram 43] FIG. 13 is a diagram showing an example of a state in which the smartphone is slightly moved in the movement direction. [Diagram 44] FIG. 13 is a diagram showing an example of a state in which a message is displayed instructing the smartphone to be moved to a predetermined position and then stopped. [Diagram 45] 45A and 45B are diagrams showing an example of image information before and after correction when shooting is performed in the state shown in FIG. 44. [Diagram 46] FIG. 45(b) is a diagram showing an example of a state in which an object detection process has been performed on the image information of FIG. 45(b). [Figure 47] FIG. 13 is a diagram showing an example of a state in which a moving display is superimposed. [Figure 48] FIG. 13 is a diagram showing an example of a state in which the smartphone is slightly moved in the movement direction. [Figure 49] FIG. 13 is a diagram showing an example of a state in which a message is displayed instructing the smartphone to be moved to a predetermined position and then stopped. [Figure 50] 50A and 50B are diagrams showing an example of image information before and after correction when shooting is performed in the state shown in FIG. 49. [Figure 51] FIG. 50(b) is a diagram showing an example of a state in which an object detection process has been performed on the image information of FIG. [Figure 52] FIG. 13 is a diagram showing an example of a state in which a moving display is superimposed. [Diagram 53]FIG. 13 is a diagram showing an example of a state in which the smartphone is slightly moved in the movement direction. [Figure 54] FIG. 13 is a diagram showing an example of a state in which a message is displayed instructing the smartphone to be moved to a predetermined position and then stopped. [Figure 55] 55A and 55B are diagrams showing an example of image information before and after correction when shooting is performed in the state shown in FIG. 54. [Figure 56] FIG. 55(b) is a diagram showing an example of a state in which an object detection process has been performed on the image information of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0057] A block diagram of an example of the overall processing functions of an information processing system 1 of the present invention is shown in Fig. 1. The information processing system 1 preferably functions in a portable communication terminal 2 such as a smartphone having a function of an image capturing device 75.

[0058] The portable communication terminal 2 is an information processing device (computer) equipped with the functions of the photographing device 75 that can be carried around, such as a smartphone or a tablet computer. It may also be a laptop computer. As long as it has the functions necessary for the processing of the information processing system 1 of the present invention, the appearance and the name of the device are not important.

[0059] 2 shows an example of a hardware configuration of the portable communication terminal 2 in the information processing system 1. The portable communication terminal 2 has a calculation device 70 such as a CPU that executes calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 that allows information to be input, a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN, an imaging device 75 such as a camera, and various sensor devices 76.

[0060] When the portable communication terminal 2 is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals 2 such as tablet computers and smartphones, but are not limited thereto.

[0061] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display by a predetermined input device (such as a touch panel pen) or a finger.

[0062] The image capturing device 75 may be any device that captures image information using visible light or the like.

[0063] The sensor device 76 may include a sensor that detects the angle of view captured by the imaging device 75, a sensor that detects the pitch angle and roll angle of the portable communication terminal 2 or the imaging device 75, a distance measuring sensor that measures the distance from the imaging device 75 to the object to be photographed, and the like. It is desirable for the sensor device 76 to be equipped with some or all of these, but it is not limited to these.

[0064] The means in the present invention are only logically distinguished in function, and may be physically or practically in the same area. The order of processing in each means of the present invention may be changed as appropriate. Also, part of the processing may be omitted. For example, the processing for determining the viewpoint direction, which will be described later, may be omitted. In that case, processing may be performed on image information that has not been processed to determine the viewpoint direction.

[0065] In the information processing system 1, functions are distributed between the management server 3 and the portable communication terminal 2. The way in which these functions are distributed can be changed as appropriate.

[0066] The portable communication terminal 2 in the information processing system 1 has an image capture processing unit 20, an image information storage unit 21, a correction processing unit 22, an object detection processing unit 23, an out-of-view determination processing unit 24, an image selection processing unit 25, and an upload processing unit .

[0067] The photography processing unit 20 photographs an object to be photographed, for example, a display shelf in a store, using the photography device 75. When photography starts, the photography processing unit 20 detects that the angle of view has moved a certain distance by a sensor that detects the angle of view in the sensor device 76, and then photographs the object. This is repeated until photography is completed.

[0068] For example, a person in charge of taking pictures fixes the portable communication terminal 2 in a fixed position while pointing the camera device 75 of the portable communication terminal 2 in the direction of a display shelf, and changes the orientation of the camera device 75 of the portable communication terminal 2 so that the camera device 75 takes pictures from above to below the display shelf. The camera processing unit 20 takes pictures when the image capture starts, and then when a sensor that detects the angle of view of the sensor device 76 detects that the angle of view has moved a fixed distance, for example, 1 / 10 of the vertical distance of the image information, the image capture processing unit 20 takes pictures with the camera device 75. This is repeated until the image capture is completed.

[0069] This allows the image capturing and processing unit 20 to capture a plurality of consecutive image information pieces from the top to the bottom of the display shelf, with the vertical positions of the angle of view shifted by 1 / 10 each, for example.

[0070] The image information captured by the image capture processing unit 20 is stored in the image information storage unit 21.

[0071] The image information storage unit 21 stores image information captured by the image capture processing unit 20.

[0072] The correction processing unit 22 performs a correction process on the image information captured by the image capturing processing unit 20 and stored in the image information storage unit 21 so as to position the image in a right-facing position. The correction processing unit 22 may perform any process as long as it can correct the image information so as to position the image in a right-facing position. For example, it may perform a known keystone correction process.

[0073] When photographing a display shelf in a store with the photographing processing unit 20, it may not be possible to ensure a sufficient distance between the display shelf to be photographed and the photographing device 75 due to circumstances such as other display shelves being located behind the photographer and narrow aisles. In addition, since the display shelf is located above or below the photographer's line of sight, photographing is often performed at an elevation or depression angle. As a result, it is not possible to photograph from a position facing directly. The correction processing unit 22 corrects the photographed image information so that it is from a position facing directly, as if it were photographed from a position facing directly.

[0074] The correction processing unit 22 may generate image information by performing correction processing on the image information captured by the image capturing processing unit 20, or may output parameters (correction parameters) for transforming the image information so that it is in a facing position. For example, the correction processing unit 22 may output parameters (parameters (correction parameters) used for projective transformation) for generating image information corrected by projective transformation of the image information before correction so that it is in a facing position.

[0075] The object detection processing unit 23 detects the area of ​​an object (target object) and its type (label) in the image information corrected by the correction processing unit 22. As a process for detecting an object from image information, a process for detecting an object in the image information can be executed by using deep learning on the image information. In this case, the image information corrected by the correction processing unit 22 (image information after correction) is input to a learning model in which the weighting coefficient between neurons in each layer of a neural network consisting of many intermediate layers is optimized, and the area and type of the object in the image information are detected. As a learning model, various image information of a display shelf is given the object and its label as correct answer data. An example of an object label and its meaning is shown in FIG. 8.

[0076] The object detection processing in the object detection processing unit 23 can use deep learning as well as compare sample data of various objects (image data of an object from one or more directions or data such as features based on the image data) with the image information corrected by the correction processing unit 22, and detect objects using the similarity.

[0077] The object detection process is not limited to these.

[0078] The cut-off determination processor 24 determines whether cut-off image information is present in the image information captured by the image capture processor 20. The cut-off determination process includes a process for determining cut-off in the vertical direction and a process for determining cut-off in the horizontal direction.

[0079] The process of determining whether the image information showing the top or bottom of the image information photographed by the photographing processing unit 20 includes a predetermined object detected by the object detection processing unit 23, for example, whether the space above the shelf (SpaceOverRack in the label in FIG. 8) or the stock product area (Stock in the label in FIG. 8) and the area below the shelf (RackBottom in the label in FIG. 8) are included. If the predetermined object is not detected, it is determined that there is image information that is cut off in the vertical direction.

[0080] An example of the process for determining whether or not the image is cut off in the vertical direction is shown in Fig. 9 to Fig. 11. Fig. 9 shows a case where a SpaceOverRack area is detected as the space above the shelf, Fig. 10 shows a case where a Stock area is detected as the stock product area, and Fig. 11 shows a case where a RackBottom area is detected as the space below the shelf.

[0081] The process of determining whether the image information captured by the image capture processor 20 is cut off in the left-right direction is performed based on the positional relationship of a predetermined object in the image information among the objects detected by the object detection processor 23 in the object detection process of the object detection processor 23. For example, the determination can be made based on whether a stock product area (Stock in the label of FIG. 8), a shelf product area (Faces in the label of FIG. 8), a price display area for hanging displays (HangingShelf in the label of FIG. 8), a hanging product area (HangingFaces in the label of FIG. 8), etc. are detected at a certain size or more in the left-right center of the image information (however, these objects that exist within a certain range from the right or left end are excluded from the determination as being adjacent shelves), or whether a left-right gap of a shelf level (border in the label of FIG. 8), a vertical structural material of the shelf (pillar in the label of FIG. 8), etc. are within a certain range from the left or right end. If a predetermined object is not detected at a certain size or more in the left-right center, or if the vertical structural material of the shelf is not within a certain range from the left or right end, it is determined that there is image information cut off in the left-right direction.

[0082] When the cut-off determination processing unit 24 determines that there is cut-off image information, a predetermined warning is displayed on the display device 72 of the portable communication terminal 2, such as that the image was not captured properly or that the image was cut off, and the user is prompted to capture the image again.

[0083] The image selection processing section 25 performs an image selection process for selecting image information from the image information captured by the image capture processing section 20, which is uploaded to the management server 3 and subjected to product identification processing and / or price reading processing. That is, when the image capture processing section 20 captures image information continuously at a certain amount of movement, for example, at each amount of movement of the angle of view, as in the present invention, 10 or more pieces of image information may be captured when capturing an image of one display shelf. This is shown diagrammatically in FIG. 12. FIG. 12 shows a case where one display shelf is continuously captured with 14 pieces of image information, and the capture range of each piece of image information is shown in parentheses. Note that the amount of movement is preferably the amount of movement of the angle of view, but is not limited thereto. For example, it may be the amount of movement of the portable communication terminal 2.

[0084] As shown in Fig. 12, performing product and / or price tag identification processing on all image information photographed for one display shelf requires a large processing time and processing load, and is a waste of processing. Therefore, from the image information photographed by the photographing processing unit 20 and stored in the image information storage unit 21, image information to be uploaded to the management server 3 and subjected to product and / or price tag identification processing is selected.

[0085] The image selection process in the image selection processing unit 25 executes a process of analyzing the shelf structure for the first image information to be processed from the top or bottom direction among the image information obtained by photographing the display shelf to be processed, and sets the object detected at a predetermined position of the image information, preferably the bottom or top end, as the tracking object. Then, a process is performed to skip to image information that includes the tracking object within the angle of view and is two or more images away from the image information (not adjacent to the image information), preferably the image information that is the furthest away. The skipped image information is then selected as the image information to be processed, and the same process as described above is executed. This process is repeated until the last image information is reached. FIG. 13 shows a schematic diagram of the image selection process.

[0086] For example, as shown in Fig. 14, suppose that the photography processing unit 20 takes 18 consecutive images (each of image information (1) to (18)) of a single display shelf from bottom to top at each movement amount of a predetermined angle of view. The correction processing unit 22 performs correction processing on each of these image information, the object detection processing unit 23 performs object detection processing, and the cut-off determination processing unit 24 performs cut-off determination processing. Then, the image selection processing unit 25 uses the tracking object to select each of the image information (1), (5), (11), and (18) as the image information to be processed.

[0087] The upload processing unit 26 sends to the management server 3 the image information (image information before correction) selected by the image selection processing unit 25 and the correction parameters for performing the correction on that image information output by the correction processing unit 22 in the correction processing. When sending the image information after the correction processing to the management server 3, it is not necessary to send the correction parameters for performing the correction processing. For example, in the case of FIG. 14, the image information before correction and the correction parameters for each of the image information (1), (5), (11), and (18) are sent to the management server 3.

[0088] The management server 3 includes an image information reception processing unit 30 , a correction and reproduction processing unit 31 , and a recognition processing unit 32 .

[0089] The image information reception processing unit 30 receives image information uploaded from the upload processing unit 26 of the portable communication terminal 2 and correction parameters for performing correction on the image information output by the correction processing unit 22. For example, in the case of Fig. 14, the image information reception processing unit 30 receives pre-correction image information and correction parameters for each of the image information (1), (5), (11), and (18). When the upload processing unit 26 has uploaded corrected image information, the correction parameters for performing the correction output by the correction processing are not necessary.

[0090] The correction reproduction processing unit 31 executes a correction process on the image information before correction using the parameters when the image information reception processing unit 30 receives the image information before correction and the parameters for performing the correction. When the image information corrected by the image information reception processing unit 30 is received from a portable terminal, the correction reproduction processing unit 31 is not necessary.

[0091] The recognition processing unit 32 executes a commodity identification process and / or a price reading process on the corrected image information. The recognition processing unit 32 has a commodity identification processing unit 320 and a price reading processing unit 321.

[0092] The product identification processing unit 320 executes a process of identifying a product in the corrected image information. For example, the product may be identified using deep learning, or the product may be identified using image matching processing. As the product identification processing unit 320, various known techniques can be used.

[0093] When identifying a product using deep learning, a process may be performed to first identify the area in which the product is located (face area) and then identify which product is displayed in the face area. The face area is an area of ​​the outline of the product or a predetermined area including the product, preferably a rectangular area.

[0094] In this case, the corrected image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the area (face area) in which the product is located in the image information is identified. As a learning model, a learning model in which correct answer data is given to the area (face area) in which the product is located in various image information photographed on a display shelf can be used. When performing product identification processing using deep learning, the object detection processing unit 23 may input an area in the corrected image information labeled with "Faces" (shelf product area) or "HangingFaces" (hanging product area) (area determined to contain a product) as a processing target, and the face area identification processing using deep learning may be performed on that area. The shelf product area and hanging product area are simply referred to as product areas.

[0095] Next, the image information of the face area is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the identification information of the product shown in the image information of the face area is identified. As a learning model, it is possible to use image information of various face areas to which correct answer data is given with product identification information (product identification information) as a label. The product identification information can be any information capable of identifying a product, such as the product name or JAN code.

[0096] As another method, the identification of the face area and the identification of the product may be performed as one process. In this case, for example, the following process can be executed. In this case, the corrected image information is input to a learning model in which the weighting coefficient between the neurons of each layer of a neural network consisting of many intermediate layers is optimized, and the face area shown in the image information and the identification information of the product in the face area are identified. As the learning model, various image information of a display shelf may be given correct answer data with the face area and the product identification information as labels. The product identification information may be any information that can identify the product, such as the product name and JAN code. In addition, when performing the product identification process using deep learning, the object detection processing unit 23 may input the area (product area) labeled with "Faces" or "HangingFaces" in the corrected image information as the processing target, and the object detection processing unit 23 may perform the identification process of the face area and the product identification information on the area using deep learning.

[0097] Furthermore, when image matching processing is used, sample data showing the appearance of various products (image information showing the appearance of the product from one or more directions or data such as feature quantities based thereon) may be compared with the corrected image information, and the product may be identified using the similarity. When image matching processing is performed, the object detection processing unit 23 may perform image matching processing on areas (product areas) labeled with "Faces" or "Hanging Faces" in the corrected image information.

[0098] The price reading processor 321 executes a process of reading the price from the price tag in the corrected image information. For example, the price may be read using deep learning, or may be read using character recognition processing or image matching processing. As the price reading processor 321, various known technologies may be used.

[0099] When reading prices using deep learning, for example, the following processing can be executed. In this case, the corrected image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the area of ​​the price tag shown in the image information is identified. As the learning model, various image information of a display shelf is given corrective data for the area of ​​the price tag. When performing the price tag identification processing using deep learning, the object detection processing unit 23 may input the area labeled "Shelf" (shelf area with price display) or "HangingShelf" (price display area of ​​hanging display) in the corrected image information as the processing target (hereinafter, the area with price display, such as the shelf area with price display and the price display area of ​​hanging display, is simply referred to as the price display area), and the price tag identification processing using deep learning may be performed on the area.

[0100] Furthermore, when image matching processing is used, sample data showing the appearance of various price tags (image information showing the appearance of a price tag from one or more directions or data such as features based thereon) may be compared with the corrected image information, and the price tag area may be identified using the degree of similarity. When image matching processing is performed, the object detection processing unit 23 may perform image matching processing on areas labeled with "Shelf" or "Hanging Shelf" in the corrected image information (price display areas determined to contain a price tag).

[0101] Character recognition is then performed on the area of ​​the price tag identified using deep learning, image matching processing, etc., to read the price.

[0102] The price reading process of the price reading processing unit 321 may include identifying the area of ​​the price tag and then performing character recognition processing on that area, as well as inputting corrected image information and reading the price of the price tag shown in the image information. As a learning model, a model in which the prices in the areas of price tags in various image information photographed of display shelves are given as corrective data can be used. When performing price reading processing using deep learning, the object detection processing unit 23 may input an area labeled "Shelf" or "HangingShelf" in the corrected image information (a price display area determined to contain a price tag) as a processing target, and perform price reading processing using deep learning on that area. EXAMPLES

[0103] Next, an example of processing using the information processing system 1 of the present invention will be described with reference to the flowcharts of Figures 3 to 7. In this embodiment, a case will be described in which the information processing system 1 of the present invention is used to identify products and read their prices for image information obtained by photographing a display shelf in a store with a camera (photographing device 75) built into a smartphone (portable communication terminal 2).

[0104] First, a person taking a photograph of a store's display shelves performs a predetermined operation on the store's display shelves (subject to be photographed) using the camera of a smartphone that the person has, and the photographing processing unit 20 executes the photographing process (S100).

[0105] The photographer performs a predetermined operation to start a predetermined application software stored in the smartphone. This startup causes the photography processing unit 20 to start the camera of the smartphone. Then, while pointing the photographing device 75 of the portable communication terminal 2 toward the display shelf, the photographer holds the portable communication terminal 2 in a fixed position so that the photographing position does not change as much as possible, and starts photographing from above to below, or from below to above, the display shelf (S200). That is, when the photography processing unit 20 detects that a photographing start operation has been performed so that the top of the display shelf (the space above the display shelf or the space where the stock products are placed) or the bottom of the display shelf (the floor surface on which the display shelf is placed) is captured, it performs the first photographing. Then, the photographer moves the photographing direction of the portable communication terminal 2 in a fixed direction (upward or downward) (S210). As the photographing direction of the portable communication terminal 2 is changed, the photographing processing unit 20 automatically photographs (S230) when the sensor for detecting the angle of view in the sensor device 76 detects that the angle of view has moved a certain distance, for example, 1 / 10 (S220). The photographing processing unit 20 repeats this process until a predetermined operation, for example, an operation to end photographing, is received (S240). This allows continuous image information of the display shelf to be photographed upward or downward from the fixed position of the portable communication terminal 2.

[0106] The person in charge of taking the photographs should move the smartphone from top to bottom or bottom to top so that the photographing position does not change as much as possible. In addition, for the continuously photographed image information, it is preferable that identification information is assigned sequentially from the first photographed image information to the last photographed image information so that the photographing order of each image information can be specified.

[0107] The photographing processor 20 may also display instruction information such as the photographing direction to the person in charge of photographing by performing the following display process. For example, the photographing processor 20 of the portable communication terminal 2 displays the moving distance in the next angle of view by the length of an arrow on the display device 72 of the portable communication terminal 2, and when the person in charge of photographing changes the photographing direction of the portable communication terminal 2 in the direction of the arrow, calculates the moving distance in the angle of view due to the change in the photographing direction, and changes the display of the arrow length so that the arrow becomes shorter. Then, when the length of the arrow becomes 0, the photographing processor 20 ends the display of the arrow, or displays a message such as "photographing is over" on the display device 72, so that the person in charge of photographing stops changing the photographing direction of the portable communication terminal 2. The photographing processor 20 ends the photographing process when it detects that the portable communication terminal 2 has stopped. In this case, the entire display shelf from top to bottom or bottom to top can be photographed in one or more images.

[0108] By performing the above-mentioned photographing process by the photographing processing unit 20, it is possible to photograph a plurality of consecutive pieces of image information with a certain degree of shift in the moving distance of the angle of view from the top to the bottom or from the bottom to the top of the display shelf. The image information photographed by the photographing processing unit 20 is stored in the image information storage unit 21. This is shown typically in FIG. 12 or FIG. 14. FIG. 12 shows a case where 14 consecutive pieces of image information are photographed while changing the photographing direction of one display shelf from the top to the bottom. FIG. 14 shows a case where 18 consecutive pieces of image information are photographed while changing the photographing direction of one display shelf from the bottom to the top.

[0109] Then, the correction processing unit 22 executes a correction process for correcting the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21 so that the image information is in a position facing the camera (S110). The correction process by the correction processing unit 22 may simply generate image information in which the image information itself has been corrected, or may also output parameters for correcting the image information so that the image information is in a position facing the camera. When the parameters are also output, the amount of data to be uploaded at the time of uploading can be reduced by uploading the image information before correction and the parameters in the upload process described later, as compared with the case of uploading the corrected image information. For example, FIG. 15 shows an example of image information obtained by correcting the image information of (1) in FIG. 14. FIG. 15(a) shows the image information before correction of (1) in FIG. 14, and FIG. 15(b) shows the image information obtained by correcting the image information of (1) in FIG. 14 by the correction processing unit 22. The correction process by the correction processing unit 22 is preferably performed on the image information captured by the image capture processing unit 20, but may be performed on only a part of the image information, not all of it.

[0110] After the correction processing is performed by the correction processing unit 22, the object detection processing unit 23 detects the area and type of the object appearing in the image information corrected by the correction processing unit 22 (S120). That is, the image information corrected by the correction processing unit 22 is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and objects appearing in the image information are detected.

[0111] Further, the cut-off determination processing unit 24 performs cut-off determination processing on the corrected image information, and determines whether the image information captured by the image capturing processing unit 20 contains cut-off image information (whether there is an error) (S130).

[0112] The cut-off determination processing unit 24 executes the process of determining cut-off in the vertical direction and the process of determining cut-off in the horizontal direction as described above.

[0113] As shown in Figures 9 to 11, the vertical cut-off determination process in the cut-off determination processing unit 24 determines whether a specific object has been detected in the object detection processing in the object detection processing unit 23 in the image information captured by the shooting processing unit 20, and whether it includes, for example, the space above the shelf (SpaceOverRack in the label in Figure 8) or the stock product area (Stock in the label in Figure 8), and the area below the shelf (RackBottom in the label in Figure 8).

[0114] The cut-off determination processing unit 24 performs cut-off determination processing in the left-right direction based on the positional relationship in the image information of a predetermined object among the objects detected in the object detection processing unit 23 by the object detection processing unit 23 in the image information photographed by the photographing processing unit 20. For example, it can be determined based on whether a stock product area (Stock in the label in FIG. 8), a shelf product area (Faces in the label in FIG. 8), a price display area for hanging displays (HangingShelf in the label in FIG. 8), a hanging product area (HangingFaces in the label in FIG. 8), etc. are detected at a certain size or larger in the left-right center of the image information (however, these objects that exist within a certain range from the right end or the left end are excluded from the determination as being adjacent shelves), or whether a cut-off between the left and right shelves (border in the label in FIG. 8) or a vertical structural material of the shelf (pillar in the label in FIG. 8) is within a certain range from the left or right end.

[0115] As described above, when the cut-off image determination processing unit 24 determines that there is cut-off image information (there is an error) (S130), the cut-off image determination processing unit 24 displays a predetermined warning on the display device 72 of the portable communication terminal 2, such as that the image was not captured properly or that the image was cut off, and urges the user to capture the image again. On the other hand, when the cut-off image determination processing unit 24 determines that there is no cut-off image information (there is no error), it causes the image selection processing unit 25 to execute image selection processing (S140).

[0116] The image selection processing unit 25 performs an image selection process for selecting, from the image information captured by the image capture processing unit 20, image information to be uploaded to the management server 3 and subjected to a product identification process and / or a price reading process.

[0117] The image selection processing unit 25 initializes a shelf structure table that indicates what areas the display shelf is made up of, and sets a parameter p to an initial value p=0 (S400). Then, the image information with identification information indicating that it was photographed first is selected as the processing target (S410). For example, if the photographing processing unit 20 photographs images from above to below, the image information photographing the top, that is, the image information having identification information of "#0" in Fig. 13, is selected as the processing target.

[0118] The image selection processing unit 25 executes a shelf structure analysis process for the image information selected as the processing target (S420). The shelf structure analysis process is a process for detecting and associating the areas where products are placed on the display shelf with the areas of the price tags, and for example, executes the process of the flowchart in FIG.

[0119] In the shelf structure analysis process, first, the image information of the parameter p is added as selected image information (S500). That is, the image information to be processed, in this case, the image information of "#0", is added as image information selected by the image selection processing unit 25.

[0120] Then, for the image information to be processed, i.e., the image information of "#0", objects detected by the object detection processing unit 23 are extracted (S510). This is shown diagrammatically in Fig. 13(a). In Fig. 13(a), "Stock", "ShelfWoPC", "Faces", and "Shelf" are detected as objects, and these are extracted.

[0121] At this time, objects on the adjacent shelf are excluded (S520). In Fig. 13(a), since there are no objects on the adjacent shelf, the process is executed as is.

[0122] The objects are then sorted in the order of the direction in which they were photographed (S530). In FIG. 13, the photographs were taken from top to bottom, so the objects are sorted from top to bottom. In the case of FIG. 13(a), the objects are sorted in the order of "Stock," "ShelfWoPC," "Faces," and "Shelf." The sorted objects are then written into a shelf structure table (S540). FIG. 16 shows an example of a shelf structure table for the image information to be processed (image information of #0).

[0123] Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560). For example, the image selection processing unit 25 selects an action for an object based on the rule, an example of which is shown in Fig. 17, and executes processing to create pairs.

[0124] Fig. 17(a) is a table showing the names of actions to be performed based on the relationship between the current object and the object one level below, and Fig. 17(b) is a table associating action names with the contents of the actions. The rules in Fig. 17 are merely examples and are not limiting. It is preferable to have rules that associate the relationship between adjacent objects, and action names with the contents of the actions, but any rules may be used.

[0125] In the case of the shelf structure table of FIG. 16, first, the topmost object is "Stock" and the object immediately below it is "ShelfWoPC", so the table of FIG. 17 is referenced to select an action with the action name "pairing", that is, these two objects are paired. Then, the next object is "ShelfWoPC" and the object immediately below it is "Faces", so the table of FIG. 17 is referenced to select an action with the action name "OK", that is, do nothing. Then, the next object is "Faces" and the object immediately below it is "Shelf", so the table of FIG. 17 is referenced to select an action with the action name "pairing", that is, these two objects are paired. The next object is "Faces", but there is no object immediately below it, so it is determined that the action selection process for all objects in the image information to be processed has ended (S550), and the shelf structure analysis process is terminated.

[0126] When the image selection processing unit 25 ends the shelf structure analysis processing, if the image information for which the shelf structure analysis processing has been performed is not the last image information (S430), it executes skip processing to select a skip destination image (S440).

[0127] The skip process is a process of finding other image information including a tracking object included in the image information to be processed, that is, a process of finding image information to be skipped. For example, the object at the bottom of the image information to be processed is set as a tracking object, and other image information that includes the area of ​​the tracking object at the top is found, and the found image information is set as the skip destination image information to be processed next.

[0128] In the skip process, a tracking object is determined in the image information (image information #0) to be processed by parameter p, and its area is recorded (S600). For example, the bottommost object in image information #0, the "Shelf" object, is set as the tracking object, and its area is recorded.

[0129] The parameter p is incremented to p=1 (S610). Since it is not yet the last image information (S620), the image information of the identification information next to the image information of the identification information of #0, i.e., the image information of the identification information of #1 is selected, and the area of ​​the image information recorded in S600 is obtained (S630). At this time, an image matching process may be performed between the image information of the tracking object area recorded in S600 and the image information of #1 to determine whether there is an area similar to the tracking object area. Furthermore, among the objects in the image information of #1, the area of ​​the object of the same type as the tracking object, "Shelf", may be matched. If the similarity is equal to or greater than a predetermined threshold, it is determined that the image information of #1 has a tracking object.

[0130] If the image information of the incremented p has an area corresponding to the tracking object (S640), p is incremented to p=2 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650). The presence of an area corresponding to the tracking object means that the image information has an area whose similarity to the tracking object area is equal to or greater than a predetermined threshold, and the absence of an area corresponding to the tracking object means that the image information has no area whose similarity to the tracking object area is equal to or greater than a predetermined threshold, for example, when there is no area corresponding to the tracking object, or when the area corresponding to the tracking object is only partially corresponding (cut off).

[0131] By repeating the above process, image information to which the image is to be skipped is searched for.

[0132] For example, when p=6 (image information of #6), if there is no area corresponding to the tracking object (if there is no area with a similarity greater than or equal to a predetermined threshold), p is decremented, p=5 is set, and the skip process is terminated (S650).

[0133] When the skip process is completed (S440), the image information with p=5, that is, the image information #5, is selected as the processing target (S410), as shown in FIG.

[0134] Then, a shelf structure analysis process is executed for the image information of p=5, the image information of #5 (S420), and the image information of #5 is added as selected image information (S500). Then, the process from S510 onwards is executed for the image information of #5 in the same manner as described above, and objects are extracted for the image information of #5 (S510), objects on the adjacent shelves are excluded (S520), and the images are sorted in order (S530). Then, the objects are written into the shelf structure table. An example of the shelf structure table for the image information of #5 is shown in FIG. 18. Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0135] After the shelf structure analysis process is performed on the image information of #5, the skip process is performed on the image information of #0 (S440).

[0136] The image selection processing unit 25 records the area of ​​the bottommost "Shelf" object as a tracking object in the image information of #5 (S600). Then, p is incremented to p=7 (S610). Since p≦14, it is not the last image information (S620), so image matching processing is performed between the image information of the tracking object area recorded in S600 (image information of the bottommost "Shelf" area in the image information of #5) and the image information of the object "Shelf" area in the image information of #7, and it is determined whether there is an area similar to the tracking object area (S630).

[0137] If the incremented image information of p=7 (image information of #7) has an area corresponding to the tracking object (S640), p is incremented to p=8 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650).

[0138] By repeating the above process, image information to which the image is to be skipped is searched for.

[0139] For example, when p=12 (image information #12), if there is no area corresponding to the tracking object (if there is no area with a similarity greater than or equal to a predetermined threshold), p is decremented, p=11, and the skip process is terminated (S650).

[0140] When the skip process is completed (S440), the image information of p=11, that is, the image information of #11, is selected as the processing target (S410), as shown in Figure 13(c).

[0141] Then, a shelf structure analysis process is executed for the image information of p=11 and the image information of #11 (S420), and the image information of #11 is added as selected image information (S500). Then, the process from S510 onwards is executed for the image information of #11 in the same manner as described above, and objects are extracted for the image information of #11 (S510), objects on the adjacent shelves are excluded (S520), and the images are sorted in order (S530). Then, the objects are written into the shelf structure table. An example of the shelf structure table for the image information of #11 is shown in FIG. 19. Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0142] After the shelf structure analysis process is performed on the image information of #11, the skip process is performed on the image information of #5 (S440).

[0143] The image selection processing unit 25 records the area of ​​the bottommost "Faces" object as the tracking object in the image information of #11 (S600). Then, p is incremented to p=12 (S610). Since p≦14, it is not the last image information (S620), so image matching processing is performed between the image information of the tracking object area recorded in S600 (image information of the bottommost "Faces" area in the image information of #11) and the image information of the object "Faces" area in the image information of #12, and it is determined whether there is an area similar to the tracking object area (S630).

[0144] If the incremented image information of p=12 (image information of #12) contains an area corresponding to the tracking object (S640), p is incremented to p=13 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650).

[0145] By repeating the above process, image information to which the image is to be skipped is searched for.

[0146] For example, when p=14, the area corresponding to the tracking object exists up to (image information #14), and if p is incremented to p=15 (S610), p≦14 is obtained, so p is considered to have passed the last image information (S620), p is decremented to p=14, and the skip process is terminated (S650).

[0147] When the skip process is completed (S440), the image information with p=14, that is, the image information #14, is selected as the processing target (S410). This is shown in Figure 13(d).

[0148] Then, a shelf structure analysis process is executed for the image information of p=14 and the image information of #14 (S420), and the image information of #14 is added as selected image information (S500). Then, the process from S510 onwards is executed for the image information of #14 in the same manner as described above, and objects are extracted for the image information of #14 (S510), objects on the adjacent shelves are excluded (S520), and the images are sorted in order (S530). Then, the objects are written into the shelf structure table. An example of the shelf structure table for the image information of #14 is shown in FIG. 20. Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0149] When the shelf structure analysis process is executed for the image information of #14, since it is the last image information (S430), the image selection processing unit 25 ends the image selection process.

[0150] By executing the above-mentioned processing, the image selection processing unit 25 can select each of the image information items #0, #5, #11, and #14 in FIG. 12 as image information to be uploaded.

[0151] After the image selection process is completed, upload processing unit 26 sends the image information (image information before correction) selected by image selection processing unit 25 and the parameters for performing correction on that image information output by correction processing unit 22 in the correction process to management server 3 (S150). That is, upload processing unit 26 sends each of the image information before correction of #0, #5, #11, and #14 and the parameters for performing correction to management server 3.

[0152] The upload processing unit 26 may send the corrected image information of #0, #5, #11, and #14 to the management server 3.

[0153] In the upload process, the upload processing unit 26 may send a shelf structure table corresponding to each piece of image information.

[0154] Then, when the image information reception processing unit 30 of the management server 3 receives each pre-correction image information and the parameters for performing the correction uploaded from the portable communication terminal 2, that is, each pre-correction image information of #0, #5, #11, and #14 and the parameters for performing the correction, the correction reproduction processing unit 31 performs a correction reproduction process on the uploaded pre-correction image information using the parameters for performing the correction process (S160). Therefore, each pre-correction image information of #0, #5, #11, and #14 is transformed into corrected image information using the parameters for performing the correction. The correction reproduction process is similar to the correction process in the portable communication terminal 2, but since it only transforms (projectively transforms) the pre-correction image information using the parameters, it can be executed by arithmetic processing and can be executed at high speed.

[0155] When the image information reception processing unit 30 receives the corrected image information, the correction reproduction processing in the correction reproduction processing unit 31 is not necessary.

[0156] Then, the product identification processing unit 320 in the management server 3 executes a product identification process on each corrected image information, and identifies the products displayed in the areas where products are displayed in each image information (areas labeled "Faces" or "Hanging Faces" (areas where it is determined that products are present)) (S170, S180).

[0157] That is, a commodity identification process is executed for each of the corrected image information of #0, #5, #11, and #14, and the commodity displayed in the commodity display area in each of the image information is identified (S170, S180).

[0158] Similarly, the price reading processing unit 321 in the management server 3 executes a price reading process on each corrected image information, and reads the price of each price tag in the area where the price tag is present in each image information (the area labeled "Shelf" or "HangingShelf" (the price display area where a price tag is determined to be present)) (S175, S185).

[0159] By executing the above-described process, appropriate photography can be performed when photographing in a store, and products displayed on display shelves can be identified.

[0160] If the shelf structure table corresponding to each image information is also uploaded in the upload process, the shelf structure table can be combined based on the tracking object to identify the structure of one display shelf. Therefore, it is possible to identify which product is displayed on which shelf and how much the price tag states based on the results of the product identification process and price reading process for each image information. EXAMPLES

[0161] In the above-mentioned first embodiment, a case has been described in which various known correction processing techniques capable of executing correction processing to obtain a straight-ahead position can be used as the correction processing in the correction processing unit 22, but a case can be described in which correction processing with a reduced processing load is realized. This allows correction processing to be executed with a reduced load even in a portable communication terminal 2 with low processing power such as a smartphone.

[0162] An example of the configuration of the correction processing unit 22 in this embodiment is shown in FIG. 21, and an example of the correction processing in the correction processing unit 22 is shown in the flowchart of FIG.

[0163] In this embodiment, the correction processing unit 22 performs correction processing by regarding the front surface of the object to be photographed as a vertical rectangle (including a square) in three-dimensional space (this surface is called the object surface) and estimating the positional relationship between the portable communication terminal 2 and the object surface in three-dimensional space using sensor information detected by the sensor device 76 of the portable communication terminal 2. An example of image information (image information before correction) photographed by the photographing device 75 is shown in Fig. 23.

[0164] The correction processing unit 22 has a sensor information input reception processing unit 220 , an angle estimation processing unit 221 , an image transformation processing unit 222 , and an output processing unit 223 .

[0165] The sensor information input reception processing unit 220 receives input of information from a sensor that detects the pitch angle and roll angle of the portable communication terminal 2 or the imaging device 75 from the sensor device 76. It also receives input of distance information between the imaging device 75 and the object to be photographed from a sensor (distance measurement sensor) that measures the distance to the object to be photographed. As the distance information, distance information from the imaging device 75 to each point on the object to be photographed is obtained.

[0166] The angle estimation processing unit 221 converts the distance information to each point of the object to be photographed, which is received by the sensor information input reception processing unit 220, into three-dimensional information, and rotates this point group based on the roll angle and pitch angle. Since the object surface is vertical, the result of the rotation is a plane that is approximately perpendicular to the xz plane, as shown in Fig. 24. Then, by calculating a regression line of these points on the xz plane, the perpendicular line drawn from the imaging device 75 to this line is estimated as the yaw angle.

[0167] The image transformation processing unit 222 uses the yaw angle estimated by the angle estimation processing unit 221 to transform the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21 so that the image information is positioned directly facing the image capture device 75. The shape of the target rectangular surface is determined by assuming four vectors with the image capture device 75 as end points, which represent the angle of view of the image capture device 75 in three-dimensional space, and then rotating the vectors based on the roll angle and pitch angle inputted by the sensor information input reception processing unit 220, and then calculating four intersections with a vertical plane tilted to the left and right by the yaw angle estimated by the angle estimation processing unit 221. Then, the image information captured by the image capture processing unit 20 is subjected to a projective transformation process such as OpenCV using the calculated four intersections, thereby performing an image transformation process, and a correction process can be performed on the image information from the directly facing position. FIG. 25 shows an example of image information corrected from the image information of FIG. 23.

[0168] The output processing unit 223 outputs the image information corrected by the correction processing unit 22. For example, the output processing unit 223 outputs the image information corrected as shown in Fig. 25. The output processing unit 223 may also output parameters for correction by the correction processing unit 22, i.e., parameters for performing projective transformation processing.

[0169] Next, the correction process of the correction processor 22 in the second embodiment will be described with reference to the flowchart of FIG.

[0170] First, a photographer photographs the store's display shelves (subject to be photographed) using the camera of the smartphone he or she carries. The photography processing unit 20 stores the image information photographed by the smartphone camera in the image information storage unit 21 (S700). An example of this image information is shown in FIG. 23. The image information in FIG. 23 was photographed from a diagonally downward direction looking upward (elevation angle) of the display shelves, and the shelves are slightly rotated to the left (pitch angle is 18.8 degrees upward, roll angle is 4.6 degrees counterclockwise).

[0171] In addition, the sensor information input reception processing unit 220 receives input of information on the pitch angle and roll angle of the smartphone or camera when capturing image information with the photographing device 75, as measured by the sensor device 76, and information on the distance to each point on the object to be photographed (S700).

[0172] The distance information may be obtained from any number of points, for example, from several tens to a hundred points, for the object to be photographed. Note that the number of points from which the distance information is obtained may be any number as long as sufficient accuracy can be obtained.

[0173] The angle estimation processing unit 221 plots the distance information received by the sensor information input reception processing unit 220 in a three-dimensional space, and converts the distance information into a point cloud in the three-dimensional space (S710). Then, the angle estimation processing unit 221 rotates this point cloud using the pitch angle and roll angle information received by the sensor information input reception processing unit 220 (S720).

[0174] In order to generate corrected image information (keystone correction image information) in which the products displayed on the display shelves on the target surface are upright and the shelves are horizontal, that is, in order to correct the target surface of the image information captured by the image capture processing unit 20 to a position facing directly, it is necessary to calculate how much the target surface is tilted from the direction facing directly to the camera. For this reason, the point cloud after the rotation in S720 is mapped onto a horizontal plane (xz plane), and a regression line is calculated using a robust estimation algorithm such as RANSAC (S730). FIG. 26 is a front view (xy plane in FIG. 24) of the point cloud after the rotation in S720 mapped onto a vertical plane, and FIG. 27 is a top view (xz plane in FIG. 24) of the point cloud after the rotation in S720 mapped onto a horizontal plane. FIG. 28 also shows a schematic diagram of the processing in the angle estimation processing unit 221.

[0175] Then, the angle (yaw angle) of the target surface from the optical axis of the camera can be calculated from this regression line. Specifically, if the slope of the regression line of the point cloud is a, the slope a of the regression line is converted to a yaw angle by calculating Equation 1 (S740). In other words, the arctangent is calculated using the slope a as an argument. (Number 1) Yaw angle = atan(a)

[0176] Since the angle estimation processing unit 221 can estimate the yaw angle by the above-mentioned processing, the image transformation processing unit 222 uses the estimated yaw angle to transform the image information captured by the shooting processing unit 20 and stored in the image information storage unit 21 so that the image information faces directly toward the shooting device 75. This processing is shown diagrammatically in FIG.

[0177] Since the angle estimation processing unit 221 has estimated the yaw angle and the positional relationship between the camera and the target surface in three-dimensional space is now clear, the image transformation processing unit 222 sets four lines (vectors) in three-dimensional space connecting the four vertices formed by the horizontal plane, the camera position, and the camera's angle of view (S750).Then, the image transformation processing unit 222 rotates these four lines (vectors) using the roll angle and pitch angle received as input by the sensor information input reception processing unit 220 and the yaw angle estimated by the angle estimation processing unit 221 (S760).

[0178] The image transformation processing unit 222 calculates a quadrangle of a cross section obtained by cutting the four straight lines rotated in S760 with the target plane (S770). The image transformation processing unit 222 performs projective transformation on the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21 with respect to this target quadrangle to generate corrected image information (S780). The image information obtained by performing the correction processing on the image information in FIG. 23 in the correction processing unit 22 is the image information in FIG. 25.

[0179] By carrying out the above-mentioned processing, the processing of the correction processing unit 22 can be carried out.

[0180] Then, the output processing unit 223 outputs the image information corrected by the correction processing unit 22 (for example, FIG. 25). The output processing unit 223 may also output parameters for projective transformation of the target quadrangle as parameters used during correction. By outputting the parameters, the upload processing unit 26 can reproduce the correction in the correction reproduction processing unit 31 of the management server 3 by sending the image information before correction and the parameters instead of the image information after correction, thereby reducing the amount of data to be uploaded.

[0181] In addition to the above-mentioned processing by the correction processing unit 22, the following processing can also be performed.

[0182] That is, when the object to be photographed has many projections and recesses, such as a display shelf on which products are displayed, the projections and recesses of the object to be photographed have a much stronger correlation in the horizontal direction than in the vertical direction. Therefore, if the left-right direction of the photographing device 75 is slightly misaligned with the horizontal direction of the display shelf, the distance information arranged side by side is likely to be close up to the middle and far from the middle, resulting in a "step-like difference." If several such rows are mixed into the acquired distance information, the estimation result will be tilted with respect to the original object surface. This is shown diagrammatically in FIG. 30. FIG. 30(a) shows image information obtained by photographing a display shelf with many projections and recesses, for example, a display shelf on which products are displayed, as the object to be photographed. FIG. 30(a) shows that the position (point) at which the distance information is acquired may be the position of a product located in the foreground, or the position of a shelf board without products at the back. In that case, the positional relationship of the point cloud will be as shown in FIG. 30(b), and the estimation result will be easily tilted.

[0183] Therefore, in the correction process in the second embodiment, the target surface may be divided into matrices of a predetermined size, and distance information of points scattered pseudo-randomly in each matrix may be acquired. This makes it possible to randomly select points from the matrix among the points plotted with distance information when converting the distance information into a point group in a three-dimensional space in S710. This is shown diagrammatically in FIG.

[0184] Through the above processing, the accuracy of the estimation result can be improved.

[0185] Furthermore, the correction processing unit 22 may perform the following processing.

[0186] That is, when the object to be photographed is a display shelf, the person in charge of photographing usually stands near the center of the display shelf to be photographed and photographs the display shelf so that the entire left and right sides are included. In that case, usually, a part of the adjacent display shelf may be photographed near the left and right ends of the photographed image information. An example of image information showing this is shown in FIG. 32. In the case of such image information, if the left and right display shelves protrude forward or are recessed backward from the display shelf to be photographed, the regression line in S730 may be inclined. In the case of the photographed image information shown in FIG. 32(a), the display shelf that is photographed on the right side of the display shelf to be photographed protrudes forward. Therefore, when the distance information is plotted in a three-dimensional space, an abnormal value occurs on the right side as shown in the top view (xz plane) of FIG. 32(b). If these point clouds are directly processed, it affects the inclination of the regression line.

[0187] Therefore, in the correction process in the correction processor 22 of the second embodiment, when a regression line is calculated using a process such as RANSAC, the following process may be executed.

[0188] First, a predetermined number of points are randomly selected from the point cloud after rotation in S720 using processing such as RANSAC to find a regression line. Then, for all point clouds, the number of points whose error from the regression line falls within a threshold is used as an evaluation function, and this operation is repeated multiple times, and the regression line with the most points is selected as the final regression line. For example, when finding a regression line from a point cloud of 100 points, a regression line is found by selecting three points as the predetermined number of points, and sufficient accuracy can be obtained by repeating the process 500 times.

[0189] In addition, this evaluation function can be modified to lower the weight when selecting points on the outer edge, thereby reducing the effect of unevenness of the display shelves reflected near the left and right edges. One way to change the weight is to count points on the outer edge as 0.5 instead of 1.

[0190] More specifically, as shown in FIG. 33, the point cloud acquisition section in the x-axis direction is divided into a predetermined number of parts, for example, four parts, and the weighting of the number of points in the outer sections is changed according to their location on the outer edge.

[0191] The correction processing in the correction processing unit 22 in the second embodiment can be applied not only to image information captured at an elevation angle or depression angle, but also to image information captured from the left and right directions. EXAMPLES

[0192] In the skip process of the image selection processing unit 25 in the first embodiment described above, the process of identifying image information to be skipped by using the process of comparing the image information in the tracking object area with similar image information has been described.

[0193] Therefore, since comparing image information imposes a heavy processing load, a case in which image information comparison processing is not used will be described in this embodiment 3. An example of processing in this case is shown in the flowchart of Fig. 34. Note that the process up to the skip processing in the image selection processing unit 25 is the same as in embodiment 1 or 2.

[0194] A tracking object is determined in the image information (image information #0) to be processed by parameter p, and the position information (coordinate information) of the rectangle that constitutes the area is recorded (S800). For example, the bottommost object in image information #0, the "Shelf" object, is determined as the tracking object, and the position information of the rectangle that constitutes the area is recorded. The position of the tracking object has been identified by the object detection process in object detection processing unit 23.

[0195] The parameter p is incremented to p=1 (S810). Since it is not yet the last image information (S820), the image information of the identification information next to the image information of the identification information of #0, i.e., the image information of the identification information of #1 is selected, and the amount of movement (movement distance) of the angle of view from the image information of the identification information of #0 to the image information of the identification information of #1 is specified (S830). Note that the amount of movement of the angle of view is the amount of movement by which the shooting processing unit 20 automatically performs shooting. For example, the amount of movement is 1 / 10 of the angle of view.

[0196] Then, the coordinate value corresponding to the amount of movement in S830 is added to the position information of the rectangle that constitutes the area of ​​the tracking object recorded in S800 (S840). This makes it possible to identify where the tracking object in the image information of #0 is located in the image information of #1.

[0197] If the rectangular area that constitutes the tracking object is included in the image information in #1, it is determined that the tracking object is present in the image information in #1.

[0198] If the image information of the incremented p contains an area corresponding to a tracking object (S850), p is incremented to p=2 (S810). On the other hand, if there is no area corresponding to a tracking object (S850), p is decremented (S860).

[0199] By repeating the above process, image information to which the image is to be skipped is searched for.

[0200] FIG. 35 is a schematic diagram showing the process of determining whether the tracking object is included in the image information indicated by the parameter p. FIG. 35(a) is a schematic diagram showing the amount of movement of the angle of view of the image information of p=n and the image information of p=n+1. One movement results in the amount of movement of the angle of view (α, β). Therefore, the position information (x, y) of a rectangle constituting the area of ​​the tracking object in the image information when p=n becomes the position information (x+α, y+β) of a rectangle constituting the area of ​​the corresponding tracking object in the image information of p=n+1. FIG. 35(b) shows this diagram. Therefore, it is sufficient that the position coordinates (x+α, y+β) of the vertices of the rectangle in the image information of p=n+1 are included in the area of ​​the image information of p=n+1. Although only one point is shown here, it is possible to determine whether the tracking object is included in the image information of p by determining that it includes four points that form a rectangle, or two points that form the bottom side (when moving from top to bottom), or two points that form the top side (when moving from bottom to top).

[0201] By carrying out the above process, it is possible to identify image information to be skipped without carrying out a process of comparing image information in the tracking object area with similar image information, thereby reducing the processing load on the image selection processing unit 25.

[0202] In addition, in the skip processing, when the corrected image information is not used, the shape and position information of the tracking object, and the position information of the image information of the parameter p may be calculated using the yaw angle identified by the correction processing unit 22, and the roll angle and pitch angle detected by the sensor device 76, and then processing may be performed. EXAMPLES

[0203] As a modified example of Examples 1 to 3, the upload processing unit 26 may be configured to, rather than uploading the corrected image information or the uncorrected image information and correction parameters, cut out and upload image information of areas in the corrected image information where it is determined that there is a product (areas labeled "Faces" or "Hanging Faces" by the object detection processing unit 23) and image information of price display areas in the corrected image information where it is determined that there is a price tag (areas labeled "Shelf" or "Hanging Shelf" by the object detection processing unit 23).

[0204] In this case, the management server 3 does not need to be provided with the correction reproduction processing unit 31 .

[0205] The product identification processing unit 320 of the management server 3 executes a product identification process similar to that of the first embodiment on the image information of an area in the corrected image information received by the image information reception processing unit 30 from the portable communication terminal 2 in which it is determined that a product is present.

[0206] In addition, the price reading processing unit 321 of the management server 3 performs a price reading process similar to that of Example 1 on the image information of the price display area in which it is determined that a price tag is present in the corrected image information received by the image information receiving processing unit 30 from the portable communication terminal 2.

[0207] This makes it possible to further reduce the amount of image information data to be uploaded. EXAMPLES

[0208] In the above-mentioned photographing processing unit 20 of the first to fourth embodiments, a case has been described in which one display shelf is continuously photographed with a plurality of pieces of image information, but if the aisle width is sufficiently wide and one display shelf can be photographed with one piece of image information, the photographing processing unit 20 may photograph one display shelf with one piece of image information. In this case, it is not necessary to perform image selection processing in the image selection processing unit 25. That is, the correction processing in the correction processing unit 22 of the first or second embodiment is performed on the image information photographed by the photographing processing unit 20, and the object detection processing in the object detection processing unit 23 is performed on the corrected image information.

[0209] After the cut-off determination processing is performed in the cut-off determination processing unit 24, the upload processing unit 26 uploads the corrected image information or the uncorrected image information and the correction parameters. Then, in the management server 3, the correction reproduction processing unit 31 performs the correction reproduction processing, the product identification processing unit 320 performs the product identification processing, and the price reading processing unit 321 performs the price reading processing.

[0210] Furthermore, when performing the same processing as in the fourth embodiment, after the cut-off determination processing is performed in the cut-off determination processing unit 24, the upload processing unit 26 uploads the image information of the area in the corrected image information where it is determined that there is a product and the image information of the price display area where it is determined that there is a price tag. Then, in the management server 3, the product identification processing unit 320 performs the product identification processing, and the price reading processing unit 321 performs the price reading processing. EXAMPLES

[0211] A modified example of the photographing processing unit 20 will be described. In the photographing processing unit 20 of the first embodiment, when photographing is started, the person in charge of photographing moves the portable communication terminal 2 in a certain direction, and when the angle of view moves a certain distance, the sensor for detecting the angle of view in the sensor device 76 detects that the angle of view has moved a certain distance and photographs the image, and this is repeated until photographing is completed. In this case, if the person in charge of photographing moves the photographing direction of the portable communication terminal 2 quickly, blurring may occur when photographing with the photographing device 75, and it may not be possible to photograph the image to an extent that allows analysis of image information.

[0212] Therefore, in the photographing processing unit 20 of this embodiment, a case will be described in which, in addition to the photographing processing in the photographing processing unit 20, the photographing processing unit 20 includes the object detection processing unit 23 and the cut-off determination processing in the cut-off determination processing unit 24. An example of a block diagram of the configuration of the information processing system 1 in this case is shown in FIG.

[0213] In the photographing processing unit 20 of this embodiment, after photographing with the photographing device 75, the correction processing unit 22 performs correction processing on the photographed image information, and the object detection processing unit 23 performs object detection processing to detect objects. Then, among the objects in the image information, a marker is set on an object in the direction of the movement of the portable communication terminal 2, preferably on the object at the end of the movement of the portable communication terminal 2, or in its vicinity. For example, when the person in charge of photographing moves the portable communication terminal 2 from top to bottom, a marker is set on an object in the lower direction, preferably on the object closest to the bottom end, among the objects detected in the image information, and when the person in charge of photographing moves the portable communication terminal 2 from bottom to top, a marker is set on an object in the upper direction, preferably on the object closest to the top end, among the objects detected in the image information, or in its vicinity.

[0214] Then, the amount of movement, for example the amount of movement of the angle of view, from an arbitrary point in the image information, for example an arbitrary point (reference point) near the end on the opposite side to the direction in which the portable communication terminal 2 is moved, to the marker is calculated, and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the display device 72 of the portable communication terminal 2. Since the direction of this arrow can specify the direction in which the portable communication terminal 2 is moved depending on the direction from the reference point to the marker, it is preferable to set the direction of the arrow as the movement direction. That is, the image capture processing unit 20 displays (movement display) such as an arrow, superimposed on the image information captured by the image capture device 75 of the portable communication terminal 2, on the display device 72, indicating the direction and / or amount of movement in which the portable communication terminal 2 is moved.

[0215] When the photographer moves the portable communication terminal 2 in the direction of the arrow, the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the movement distance to the destination. As a result, the length of the arrow becomes shorter when the photographer moves the direction of the portable communication terminal 2, and when the marker reaches an arbitrary point in the original image information, for example, an arbitrary point (reference point) near the end of the image information on the opposite side to the direction in which the portable communication terminal 2 is moved, the length of the arrow becomes 0. At that time, a movement display is displayed indicating that the portable communication terminal 2 is stopped, and the photography is automatically performed by the photography device 75. Note that, although it is preferable that the photography is automatically performed by the photography device 75, the photography operation may be performed by the photographer, and the photography processing unit 20 may accept the operation.

[0216] To automatically capture images, the camera 75 captures images after it is confirmed that the person in charge of capturing images has stopped the portable communication terminal 2 and that the acceleration detected by the sensor device 76 of the portable communication terminal 2, for example an acceleration sensor, is 0 or within a range of acceleration that can be determined as a stop. After capturing images with the camera 75, a display is displayed indicating the start of movement, for example, a movement amount of the angle of view from the reference point to the next marker is calculated and a movement display such as an arrow with a length proportional to the amount of movement is displayed. This makes it possible to prevent blurring and the like.

[0217] The above process is repeated until the entire display shelf has been photographed. By performing the above process, the number of pieces of image information photographed by the photographing processor 20 can be reduced, and the process by the image selection processor 25 becomes unnecessary.

[0218] FIG. 37 is a schematic diagram showing the processing of this embodiment. FIG. 37(a) is a side view of the state in which the photographer is photographing the display shelf, and FIG. 37(b) is a front view of the state in which the photographer is photographing the display shelf. Also, p1 to p4 are the positions of the portable communication terminal 2 when the display shelf is photographed by the photographing device 75 of the portable communication terminal 2. Note that the photographing positions of the portable communication terminal 2 are shown shifted from one another in FIG. 37 for ease of understanding, but in reality, the photographing positions of the portable communication terminal 2 may be the same or shifted, with only the photographing direction of the portable communication terminal 2 changing.

[0219] The processing in this embodiment will be described with reference to the flowcharts in Fig. 38 and Fig. 39. Fig. 38 is an example of a flowchart showing the overall processing in this embodiment, and Fig. 39 is an example of a flowchart showing the photographing processing. In addition, a case where a smartphone is used as the portable communication terminal 2 will be described. As in the case of Fig. 37, a case where the photographing direction is changed from a downward direction to an upward direction of the display shelf using the portable communication terminal 2 will be described. That is, a case where the photographing direction is changed to p1, p2, p3, and p4 will be described.

[0220] First, as in the first embodiment, a person taking pictures of the store's display shelves performs a predetermined operation on the store's display shelves (subject to be photographed) using the camera of his or her smartphone, and the photography processing unit 20 executes the photography process (S900).

[0221] The person in charge of taking pictures performs a predetermined operation to start up a predetermined application software stored in the smartphone. This start-up causes the photography processing unit 20 to start up the camera of the smartphone. Then, the person in charge of taking pictures holds the smartphone at a certain position so that the shooting position of the smartphone does not change as much as possible while pointing the camera of the smartphone toward the display shelf, and starts the shooting process from above to below or from below to above the display shelf. That is, the photography processing unit 20 repeats the process from S1000 onwards until the shooting end condition is met, such as the top of the display shelf (the space above the display shelf or the space where stock products are placed) or the bottom (the floor surface on which the display shelf is placed) being captured. Here, since the direction of the smartphone to shoot is changed from the bottom to the top, the shooting end condition is when an object at the top of the display shelf (the space above the display shelf or the space where stock products are placed) is recognized.

[0222] 37, since shooting starts from the bottom, the person in charge of shooting points the smartphone in the direction of p1, which is the shooting start position. When the shooting processing unit 20 detects that the smartphone has stopped to a certain extent using an acceleration sensor of the sensor device 75 (S1000), it takes the first shot (S1010). At this time, the shooting processing unit 20 may automatically take the shot with the shooting device 75, or may issue an instruction to press the shutter of the smartphone camera and have the person in charge of shooting take the shot.

[0223] For example, Fig. 40(a) shows an example of image information captured when the first image is captured so that the bottom part is captured. Fig. 40(a) shows the state when the display shelf is captured from position p1 so that the floor surface is captured.

[0224] The image information captured by the image capture processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 executes a correction process to correct the image to a frontal position (S1020). The process may be a known keystone correction process as in the first embodiment, or a correction process as in the second embodiment. If the image capture processing unit 20 captures the image from a frontal position, the correction processing unit 22 does not need to perform the correction process. Figure 40(b) shows the image information after the correction process.

[0225] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing unit 22 (S1030).

[0226] That is, the area of ​​an object (target object) and its type (label) are detected from the image information captured by the capture processing unit 20 or the image information corrected by the correction processing unit 22. The object detection processing in the object detection processing unit 23 can be performed in the same manner as in each of the above-mentioned embodiments. FIG. 41 shows the state in which the area and type of an object are detected from the image information after the correction processing in FIG. 40(b). FIG. 41 shows the state in which the area and type of each of the objects "RackBottom", "Shelf", "Faces", and "Shelf" are detected from the bottom.

[0227] The photographing processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether a photographing end condition, for example, a predetermined object at the top or bottom of the photographing direction, such as "SpaceOverRack" or "RackBottom", has been detected (S1040). Here, since "SpaceOverRack", which indicates that photographing has been completed up to the top of the display shelf, has not been detected, the photographing processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction in which the smartphone is moved to photograph, preferably on the object at the end or in its vicinity (S1050). Note that the object becomes a tracking object as the marker, so it may be used. In this case, since the smartphone is moved from the bottom to the top, a marker is set near the top of the object "Shelf" located at the top. Note that the marker and the tracking object may be the same, and all or part of the area of ​​the object "Shelf" located at the top of FIG. 41 may be set as the marker.

[0228] Then, the amount of movement of the angle of view from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the end opposite to the direction in which the smartphone is moved, to the marker is calculated (S1060), and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the display of the smartphone (S1070). Since the direction of this arrow can specify the direction in which the smartphone is moved depending on the direction from the reference point to the marker, it is preferable to set the direction of the arrow as the movement direction. That is, the image capture processing unit 20 displays (movement display) such as an arrow on the display of the smartphone, superimposed on the image information captured by the smartphone camera, indicating the direction and / or amount of movement of the smartphone.

[0229] FIG. 42 shows a schematic diagram of this state. At this time, since the image information before correction (FIG. 40(b)) is displayed on the display of the smartphone, the image information may be superimposed on the image information. In this case, since the amount of movement of the angle of view from the reference point to the marker is calculated using the image information after correction, the amount of movement of the angle of view from the reference point to the marker in the image information before correction may be converted to the amount of movement of the angle of view from the reference point to the marker in the image information before correction, and the length of the arrow may be displayed. Alternatively, since the vertical deviation before and after correction is not large, even if the amount of movement is calculated based on the image information after correction, the tracking object or a marker provided in its vicinity does not deviate from the angle of view very much. Therefore, an arrow may be superimposed on the image information before correction with the length of the arrow corresponding to the amount of movement calculated using the image information after correction.

[0230] In either case, the image capturing processing unit 20 performs moving display of an indication indicating the direction and / or amount of movement of the smartphone, superimposed on the image displayed on the display of the smartphone.

[0231] Then, when the person in charge of photography moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the movement distance to the destination (marker). An example of this state is shown in Fig. 43. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0232] When the person taking the picture moves the direction of the smartphone, the length of the arrow becomes shorter, and when an arbitrary point in the original image information, for example an arbitrary point (reference point) near the end of the image information on the opposite side to the direction in which the smartphone is moved, reaches the position of the original marker, the length of the arrow becomes 0. At that time, a movement display is displayed indicating that the smartphone is to be stopped, and the smartphone camera automatically takes a picture. An example of this state is shown in FIG. 44.

[0233] When this message is displayed, the person in charge of photography stops the smartphone (S1000), and the photography processing unit 20 again takes a photograph with the smartphone camera in the same manner as described above (S1010). At this time, the image information taken by the photography processing unit 20 with the smartphone camera is shown in FIG. 45(a).

[0234] Then, the image information captured by the image capture processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 executes a correction process so as to achieve a position facing directly (S1020). Fig. 45(b) shows the image information after the correction process.

[0235] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing unit 22 (S1030). Figure 46 shows the state in which the object areas and types are detected for the image information after the correction processing in Figure 45(b). Figure 46 shows the state in which the areas and types of each of the objects "Shelf", "Faces", and "Shelf" are detected from the bottom.

[0236] The photographing processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether or not "Space Over Rack" has been detected, which is the photographing end condition (S1040). Here, since "Space Over Rack", which indicates that photographing has been completed up to the top of the display shelf, has not been detected, the photographing processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction in which the smartphone is moved to photograph, preferably on the object at the extreme end or in its vicinity (S1050). All or part of the area of ​​the object "Shelf" at the top of FIG. 46 is set as the marker.

[0237] Then, the amount of movement of the angle of view from an arbitrary point in the image information, for example an arbitrary point (reference point) near the end on the opposite side to the direction in which the smartphone is moved, to the marker is calculated (S1060), and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the smartphone display (S1070). This state is shown diagrammatically in FIG.

[0238] Then, when the person in charge of photography moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the movement distance to the destination (marker). An example of this state is shown in Fig. 48. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0239] When an arbitrary point (reference point) near the end of the image information on the opposite side to the direction in which the smartphone is moved reaches the position of the original marker, the length of the arrow becomes 0. At that time, a movement display is displayed indicating that the smartphone is to be stopped, and a photograph is automatically taken with the smartphone camera. An example of this state is shown in Figure 49.

[0240] When this message is displayed, the person taking the photographs stops the smartphone (S1000), and the photographing processing unit 20 takes a photograph again with the smartphone camera in the same manner as described above (S1010). At this time, the image information taken by the photographing processing unit 20 with the smartphone camera is shown in FIG. 50(a).

[0241] Then, the image information captured by the image capture processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 executes a correction process so that the image is in a facing position (S1020). Fig. 50(b) shows the image information after the correction process.

[0242] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing unit 22 (S1030). Figure 51 shows the state in which the object areas and types have been detected for the image information after the correction processing in Figure 50(b). Figure 51 shows the state in which the areas and types of each of the objects "Shelf", "Faces", "Shelf", and "Faces" have been detected from the bottom.

[0243] The photographing processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether or not "Space Over Rack" has been detected, which is the photographing end condition (S1040). Here, since "Space Over Rack", which indicates that photographing has been completed up to the top of the display shelf, has not been detected, the photographing processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction in which the smartphone is moved to photograph, preferably on the object at the extreme end or in its vicinity (S1050). All or part of the area of ​​the object "Faces" at the top of FIG. 51 is set as the marker.

[0244] Then, the amount of movement of the angle of view from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the end on the opposite side to the direction in which the smartphone is moved, to the marker is calculated (S1060), and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the smartphone display (S1070). This state is shown diagrammatically in FIG.

[0245] Then, when the person in charge of photography moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the movement distance to the destination (marker). An example of this state is shown in Fig. 53. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0246] When an arbitrary point (reference point) near the end of the image information on the opposite side to the direction in which the smartphone is moved reaches the position of the original marker, the length of the arrow becomes 0. At that time, a movement display is displayed indicating that the smartphone is to be stopped, and a photograph is automatically taken with the smartphone camera. An example of this state is shown in Figure 54.

[0247] When this message is displayed, the person taking the photographs stops the smartphone (S1000), and the photographing processing unit 20 takes a photograph again with the smartphone camera in the same manner as described above (S1010). At this time, the image information taken by the photographing processing unit 20 with the smartphone camera is shown in FIG. 55(a).

[0248] Then, the image information captured by the image capture processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 executes a correction process so as to achieve a position facing directly (S1020). Figure 55(b) shows the image information after the correction process.

[0249] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing unit 22 (S1030). Figure 56 shows the state in which the object areas and types are detected for the image information after the correction processing in Figure 55(b). Figure 56 shows the state in which the areas and types of each of the objects "Faces", "Shelf", "Faces" and "SpaceOverRack" are detected from the bottom.

[0250] Then, the photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine a photography end condition, for example, whether a specific object has been detected (S1040). In this case, since the object "SpaceOverRack" indicating that the top of the display shelf has been photographed has been detected, the processing in the photography processing unit 20 is ended.

[0251] When the photographing process is executed in the photographing processing unit 20 as described above, the upload processing unit 26 sends the photographed image information (image information before correction) and the parameters for performing correction on the image information output by the correction processing unit 22 to the management server 3 (S910). That is, the upload processing unit 26 sends the image information and the parameters for performing correction of each of Figures 40(a), 45(a), 50(a), and 55(a) to the management server 3.

[0252] The upload processing unit 26 may send the corrected image information, that is, the image information of FIG. 40(b), FIG. 45(b), FIG. 50(b), and FIG. 55(b), to the management server 3.

[0253] In the upload process, the upload processing unit 26 may send a shelf structure table corresponding to each piece of image information.

[0254] Then, when the image information reception processing unit 30 of the management server 3 receives each piece of pre-correction image information and the parameters for performing the correction uploaded from the portable communication terminal 2, the correction reproduction processing unit 31 performs a correction reproduction process on the uploaded pre-correction image information using the parameters for performing the correction process (S920). Therefore, the correction reproduction process is executed to transform each piece of pre-correction image information in Figures 40(a), 45(a), 50(a), and 55(a) into corrected image information using the parameters for performing each correction.

[0255] When the image information reception processing unit 30 receives the corrected image information, the correction reproduction processing in the correction reproduction processing unit 31 is not necessary.

[0256] Then, the product identification processing unit 320 in the management server 3 executes a product identification process on each corrected image information, and identifies the products displayed in the areas where products are displayed in each image information (areas labeled "Faces" or "Hanging Faces" (areas where it is determined that products are present)) (S930, S940).

[0257] That is, a product identification process is executed for each of the corrected image information of Figures 40(a), 45(a), 50(a), and 55(a), and the products displayed in the areas in which the products are displayed in each of the image information are identified (S930, S940).

[0258] Similarly, the price reading processing unit 321 in the management server 3 executes a price reading process on each corrected image information, and reads the price of each price tag in the area where the price tag is present in each image information (the area labeled "Shelf" or "HangingShelf" (the price display area where a price tag is determined to be present)) (S935, S945).

[0259] By executing the above-described process, appropriate photography can be performed when photographing in a store, and products displayed on display shelves can be identified.

[0260] If the shelf structure table corresponding to each image information is also uploaded in the upload process, the shelf structure table can be combined based on the tracking object to identify the structure of one display shelf. Therefore, it is possible to identify which product is displayed on which shelf and how much the price tag states based on the results of the product identification process and price reading process for each image information.

[0261] By executing the above-described processing, the processing can be executed in the same manner as in the first embodiment.

[0262] When performing the object detection process, the image capture processing unit 20 may perform an out-of-view determination process, similar to Example 1. When performing the out-of-view determination process on the corrected image information, for example, if it is determined that the left and right sides of a display shelf are out of view, a guide to step back before taking a picture may be displayed on the smartphone display.

[0263] In the first to sixth embodiments, the order of processing can be changed as desired. In addition, processing can be added or deleted without changing the gist of the present invention. Furthermore, since the information processing system 1 of the present invention can execute image correction processing with a small processing load, a portable communication terminal 2 such as a smartphone or tablet computer with low processing power is preferable as the information processing device, and although this case has been described, it goes without saying that the information processing device may be executed by a computer or server with high processing power.

[0264] In addition, any of the embodiments may be combined as appropriate. [Industrial Applicability]

[0265] By using the information processing system 1 of the present invention, it is possible to reduce the number of image information to be processed while reducing the burden of photographing. In addition, since it is possible to determine image information that is out of date at the store where the photographing is performed, it is possible to re-take the photograph on the spot, thereby preventing a decrease in work efficiency. [Explanation of symbols]

[0266] 1: Information processing system 2: Portable communication terminal 20: Shooting processing section 21: Image information storage unit 22: Correction processing unit 23: Object detection processing unit 24: Image cutoff detection processing section 25: Image selection processing section 26: Upload processing section 30: Image information reception processing unit 31: Correction reproduction processing section 32: Recognition processing section 220: Sensor information input reception processing unit 221: Angle estimation processing unit 222: Image transformation processing unit 223: Output processing unit 320: Product identification processing unit 321: Price reading processing unit 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74: Communication equipment 75: Shooting device 76: Sensor device

Claims

1. An information processing system for image information obtained by photographing a display shelf, an image capturing processing unit that detects the amount of movement and captures an image of the display shelf; a correction processing unit that performs a correction process on the image information captured by the image capture processing unit; an object detection processing unit that detects an object appearing in the corrected image information; a cut-off determination processing unit that performs cut-off determination using the presence or absence of an object detected from the image information and / or a positional relationship of the detected object, and issues a predetermined warning when it is determined that the object is cut off; an image selection processing unit that selects image information to be subjected to a product identification process and / or a price reading process from the multiple pieces of image information captured by the image capture processing unit or the image information corrected by the correction processing unit; a recognition processing unit that executes the commodity identification process and / or the price reading process on the selected image information; An information processing system comprising:

2. An information processing system for image information obtained by photographing a display shelf, an image capturing processing unit that detects the amount of movement and captures an image of the display shelf; a correction processing unit that performs a correction process on the image information captured by the image capture processing unit; an object detection processing unit that detects an object appearing in the corrected image information; an image selection processing unit that selects image information to be subjected to a product identification process and / or a price reading process from the multiple pieces of image information captured by the image capture processing unit or the image information corrected by the correction processing unit; a recognition processing unit that executes the commodity identification process and / or the price reading process on the selected image information; An information processing system comprising:

3. The image selection processing unit: An arbitrary object in the image information to be processed is set as a tracking object, selecting non-adjacent image information including the tracking object as skip destination image information and as image information to be subjected to the commodity identification process and / or the price reading process; 3. The information processing system according to claim 1 or 2.

4. The image selection processing unit: An object near an edge of the image information to be processed is set as a tracking object. Selecting image information that includes the tracking object and is two or more pieces of image information away from the image information to be processed as skip destination image information, as image information to be subjected to the commodity identification process and / or the price reading process; 4. The information processing system according to claim 3.

5. The image selection processing unit: using position information of a tracking object in the image information to be processed and the amount of movement, determining whether or not the tracking object in the image information to be processed is included in a shooting range of image information preceding the image information to be processed; 5. The information processing system according to claim 4.

6. The information processing system includes: a cut-off determination processing unit that performs cut-off determination using the presence or absence of an object detected from the image information and / or a positional relationship of the detected object; 2. The information processing system according to claim 1, further comprising:

7. The cut-off determination processing unit is If it is determined that there is cut-off image information, a predetermined warning is displayed.

7. The information processing system according to claim 6.

8. An information processing system for image information obtained by photographing a display shelf, an image capturing and processing unit for capturing an image of the display shelf; a correction processing unit that performs a correction process on the image information captured by the image capture processing unit; an object detection processing unit that detects an object appearing in the corrected image information; a cut-off determination processing unit that performs cut-off determination using the presence or absence of an object detected from the image information and / or a positional relationship of the detected object, and issues a predetermined warning when it is determined that the object is cut off; a recognition processing unit that executes the commodity identification process and / or the price reading process on the image information; An information processing system comprising:

9. The cut-off determination processing unit is Among the captured plurality of pieces of image information, image information on cut-off portions in the vertical direction is determined based on the presence or absence of a space above the shelf or a stock commodity area and an area below the shelf.

9. The information processing system according to claim 6 or 8.

10. The cut-off determination processing unit is Among the captured multiple image information, the left-right cut-off image information is determined by detecting whether an area for placing products or an area for displaying prices is detected in the left-right center with a size equal to or larger than a predetermined size, and whether left-right cuts or left-right structural materials are within a predetermined range from the left-right ends of the image information.

9. The information processing system according to claim 6 or 8.

11. The imaging processing unit includes: When it detects that the distance in the field of view has moved by a specified value or more, it automatically takes a picture.

9. The information processing system according to claim 1, 2 or 8.

12. An information processing system for image information obtained by photographing a display shelf, an image capturing and processing unit for capturing an image of the display shelf; a recognition processing unit that executes the commodity identification process and / or the price reading process on the image information, The imaging processing unit includes: Until the specified shooting end conditions are met, Correction processing is performed on the captured image information, when it is detected that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to movement of the portable communication terminal, photographing the image or accepting a photographing operation; An information processing system comprising:

13. The imaging processing unit includes: Until the shooting end condition is met using the object in the image information, A correction process is performed on the captured image information; Detecting an object appearing in the corrected image information; setting a predetermined location on an object in a moving direction of the portable communication terminal or in its vicinity among objects shown in the image information; when it is detected that a reference point in the captured image information or the corrected image information has moved to the predetermined position due to movement of the portable communication terminal, photographing the image or accepting a photographing operation; 13. The information processing system according to claim 12.

14. The imaging processing unit includes: Calculating the amount of movement from the reference point to the predetermined location, and displaying the movement according to the amount of movement and / or the direction of movement; 14. The information processing system according to claim 13.

15. The imaging processing unit includes: When it is detected that the reference point has moved to the predetermined location, a predetermined notification is given.

15. The information processing system according to claim 14.

16. The correction processing unit: a sensor information input reception processing unit that receives input of pitch angle and roll angle information of the imaging device or the information processing device and distance information to an arbitrary point on an object to be photographed; an image transformation processing unit that performs image transformation processing on the image information, The image transformation processing unit includes: calculating a regression line of a point cloud including a plurality of the points using the distance information and the pitch angle and roll angle information, thereby estimating information on a yaw angle of the imaging device or the information processing device; Using information on the estimated yaw angle, the image information is transformed so that it faces directly toward the image capture device.

13. The information processing system according to claim 1, 2, 8 or 12.

17. The image transformation processing unit includes: distance information to an arbitrary point on the object to be photographed is converted into three-dimensional information, the arbitrary point is rotated based on pitch angle and roll angle information, and a regression line of a point cloud including the arbitrary point is calculated when the point cloud is projected onto a horizontal plane, thereby estimating the yaw angle information.

17. The information processing system according to claim 16,

18. The correction processing unit: an output processing unit that outputs correction parameters for performing image transformation processing on the image information; 17. The information processing system according to claim 16, further comprising:

19. The recognition processing unit is executing the product identification process on the product area detected by the object detection processing unit, and / or executing the price reading process on the price display area detected by the object detection processing unit; 13. The information processing system according to claim 1, 2, 8 or 12.

20. The information processing system includes: an upload processing unit that uploads a part or all of the image information before correction and correction parameters for the image information selected by the image selection processing unit; a correction reproduction processing unit that reproduces the correction by using a part or all of the uploaded uncorrected image information and the correction parameters, The recognition processing unit is performing a product identification process and / or a price reading process on a part or all of the corrected image information reproduced by the correction reproduction processing unit; 13. The information processing system according to claim 1, 2, 8 or 12.

21. Computer, an image capturing processing unit that detects the amount of movement and captures an image of the display shelf; a correction processing unit that performs a correction process on the image information captured by the image capture processing unit; an object detection processing unit that detects an object appearing in the corrected image information; a cut-off determination processing unit that performs cut-off determination using the presence or absence of an object detected from the image information and / or a positional relationship of the detected object, and issues a predetermined warning when it is determined that the object is cut off; an image selection processing unit that selects image information to be subjected to a commodity identification process and / or a price reading process from the plurality of pieces of image information captured by the image capture processing unit or the image information corrected by the correction processing unit; a recognition processing unit that executes the commodity identification process and / or the price reading process on the selected image information; An information processing program characterized by causing the program to function as follows.

22. Computer, an image capturing processing unit that detects the amount of movement and captures an image of the display shelf; a correction processing unit that performs a correction process on the image information captured by the image capture processing unit; an object detection processing unit that detects an object appearing in the corrected image information; an image selection processing unit that selects image information to be subjected to a commodity identification process and / or a price reading process from the plurality of pieces of image information captured by the image capture processing unit or the image information corrected by the correction processing unit; a recognition processing unit that executes the commodity identification process and / or the price reading process on the selected image information; An information processing program characterized by causing the program to function as follows.

23. Computer, an imaging processing unit for imaging the display shelves; a correction processing unit that performs a correction process on the image information captured by the image capture processing unit; an object detection processing unit that detects an object appearing in the corrected image information; a cut-off determination processing unit that performs cut-off determination using the presence or absence of an object detected from the image information and / or a positional relationship of the detected object, and issues a predetermined warning when it is determined that the object is cut off; a recognition processing unit that executes the commodity identification process and / or the price reading process on the image information; An information processing program characterized by causing the program to function as follows.

24. Computer, an imaging processing unit for imaging the display shelves; An information processing program that causes a computer to function as a recognition processing unit that executes the commodity identification process and / or the price reading process on the image information, The imaging processing unit includes: Until the specified shooting end conditions are met, Correction processing is performed on the captured image information, when it is detected that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to movement of the portable communication terminal, photographing the image or accepting a photographing operation; 2. An information processing program comprising:

25. A portable communication terminal for use in an information processing system that photographs a display shelf and recognizes products and / or prices displayed in the image information, an image capturing processing unit that detects an amount of movement of an image capturing device of the portable communication terminal and captures an image of the display shelf; a correction processing unit that performs a correction process on the image information captured by the image capture processing unit; an object detection processing unit that detects an object appearing in the corrected image information; an image selection processing unit that selects image information to be subjected to a commodity identification process and / or a price reading process from the plurality of pieces of image information captured by the image capture processing unit or the image information corrected by the correction processing unit; an upload processing unit that uploads a part or all of the image information selected by the image selection processing unit in order to execute a product identification process and / or a price reading process using the image information; An information processing program that functions as a

26. A portable communication terminal for use in an information processing system that photographs a display shelf and recognizes products and / or prices displayed in the image information, an imaging processing unit for imaging the display shelves; An information processing program that functions as an upload processing unit that uploads a part or all of the captured image information or the corrected image information in order to execute a product identification process and / or a price reading process using the image information, The imaging processing unit includes: Until the specified shooting end conditions are met, Correction processing is performed on the captured image information, when it is detected that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to movement of the portable communication terminal, photographing the image or accepting a photographing operation; An information processing program that functions as a

Images