Information processing system

Abstract

To provide an information processing system to be used for identification / price reading of commodities displayed on a display shelf.SOLUTION: An information processing system for image information for photographing a display shelf is an information processing system including a photographing processing part for detecting a movement amount to photograph a display shelf, a correction processing part for performing correction processing to the image information photographed by the photographing processing part, an object detection processing part for detecting object reflected in the image information undergoing the correction processing, an image selection processing part for selecting image information to be an object of commodity identification processing and / or price reading processing from a plurality of pieces of image information photographed by the photographing processing part or the image information after the correct processing by the correction processing part, and a recognition processing part for executing the commodity identification processing and / or the price reading processing to the selected image information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system used to identify and read the prices of products displayed on shelves. [Background technology]

[0002] In convenience stores, supermarkets, and other stores, it is common for products to be displayed on shelves. Therefore, by displaying multiple products on a shelf, even if one product is purchased, another person can purchase the same product. Also, products displayed in large quantities in prominent locations attract attention and sell more than other products, creating a competitive advantage / inferiority relationship. Therefore, managing where and how many products are displayed on the shelf, as well as the prices of those products, is important in product sales strategies.

[0003] Therefore, by photographing the display shelves with a photographing device such as a camera and automatically recognizing the objects in the image information, it is possible to ascertain the display positions and numbers of products, the displayed products, and the prices written on the price tags (price cards).Technologies for this purpose are disclosed in Patent Documents 1 and 2 below. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 5-334409 [Patent Document 2] Japanese Patent Application Publication No. 5-342230 Summary of the Invention [Problem to be solved by the invention]

[0005] When photographing products displayed on display shelves with a camera, it is common to be restricted by various photographing conditions, such as the space available for the photographing location, the presence of obstacles, etc. For example, if the width of the aisle between the display shelves where the products are displayed is narrow, it is not possible to photograph the display shelf with a single image, and one display shelf must be photographed with multiple images.

[0006] When identifying products displayed on a display shelf, processing is performed on a shelf-by-shelf basis, so a single display shelf is constructed using multiple pieces of image information, and product identification processing and price reading processing are performed based on this.

[0007] When photographing display shelves in sections, combining multiple images into a single image results in image distortion (steps and blurring) at the combined points, which affects the accuracy of product identification and price identification. Therefore, while not combining images is better for the accuracy of product identification and price reading, it requires photographers to take as few images as possible, which increases the workload of the photographer. Therefore, it is preferable for photographers to take photographs mechanically, but this can result in an increase in the number of images taken. This increases the amount of image information to be processed, which leads to an overall increase in processing load. Therefore, there is a need to reduce the number of images to be processed while also reducing the workload of the photographer.

[0008] Furthermore, when photographing display shelves in sections, there may be gaps between the image information, resulting in areas that are not captured. Or, the left and right sides of the display shelves may be cut off. In such cases, it is necessary to take photographs again, but these situations become apparent during the actual image analysis stage. Therefore, photographs must be taken again after the image analysis stage. However, in order to take photographs, it is necessary to go back to the store and take photographs again, which significantly reduces work efficiency. [Means for solving the problem]

[0009] In view of the above problems, the present inventors have invented an information processing system for identifying products displayed on a display shelf.

[0010] The first invention is an information processing system for image information of a display shelf, which has an image processing unit that photographs the display shelf and a recognition processing unit that performs product identification processing and / or price reading processing on the image information, and the image processing unit performs correction processing on the photographed image information until a predetermined photographing end condition is met, and when it detects that a reference point in the photographed image information or the corrected image information has moved to a predetermined location due to movement of a portable communication terminal, the information processing system performs photographing or accepts photographing operations.

[0011] With the configuration of the present invention, when taking a photograph with a portable communication terminal, the photograph is taken when the reference point moves to a predetermined position, or the photographing process is performed by accepting a photographing operation, so the number of images taken can be reduced, which reduces the number of times correction processing, object detection processing, etc. are performed, and reduces the processing load.

[0012] In the above-mentioned invention, the photography processing unit can be configured as an information processing system that performs correction processing on the photographed image information until a photography termination condition is met using the object depicted in the image information, detects the object depicted in the corrected image information, sets a predetermined location at or near an object depicted in the image information that is in the direction of movement of the portable communication terminal, and performs photography or accepts photography operations when it detects that the reference point in the photographed image information or the corrected image information has moved to the predetermined location due to the movement of the portable communication terminal.

[0013] As in the present invention, a marker may be set on or near an object detected from captured image information, and processing may be performed.

[0014] In the above-mentioned invention, the photography processing unit can be configured as an information processing system that calculates the amount of movement from the reference point to the specified location and displays movement according to the amount of movement and / or the direction of movement.

[0015] By performing the processing of the present invention, the person in charge of photography can easily recognize how much and in what direction the portable communication terminal should be moved.

[0016] In the above-described invention, the photographing processing unit may be configured as an information processing system that issues a predetermined notification when it detects that the reference point has moved to the predetermined location.

[0017] As in the present invention, when it is detected that the portable communication terminal has moved to a specified location, a message is displayed indicating that the portable communication terminal should not be moved, and the person in charge of photography can operate the portable communication terminal in accordance with the instructions.

[0018] In the above-described invention, the correction processing unit includes a sensor information input reception processing unit that receives input of pitch angle and roll angle information of the image capture device or information processing device and distance information to an arbitrary point on an object to be captured, and an image transformation processing unit that performs image transformation processing on the image information, and the image transformation processing unit can be configured as an information processing system that uses the distance information and the pitch angle and roll angle information to calculate a regression line of a point cloud that includes a plurality of the points, estimates yaw angle information of the image capture device or information processing device, and transforms the image information so that it is positioned directly facing the image capture device using the estimated yaw angle information.

[0019] The configuration of the present invention enables correction processing without a large processing load, making correction processing possible even on a portable communication terminal such as a smartphone that does not have high performance.

[0020] In the above-described invention, the image transformation processing unit can be configured as an information processing system that converts distance information to an arbitrary point on a photographed object into three-dimensional information, rotates the point based on pitch angle and roll angle information, and calculates a regression line of a point cloud when the point cloud including the point is projected onto a horizontal plane, thereby estimating the yaw angle information.

[0021] It is preferable to carry out the correction process as in the present invention.

[0022] In the above-mentioned invention, the correction processing section can be configured as an information processing system having an output processing section that outputs correction parameters for performing image transformation processing on the image information.

[0023] Because the product identification process / price reading process requires a high processing load, it is preferable to perform the process on a server with high processing power rather than on a portable communication terminal with low processing power, such as a smartphone. In this case, the image information to be subjected to the product identification process / price reading process is uploaded from the portable communication terminal to the server. However, when uploading the corrected image information, blank areas are included, increasing the amount of data. Therefore, by outputting correction parameters as in the present invention, it is possible to upload the image information before correction and the correction parameters, thereby reducing the amount of data to be uploaded.

[0024] In the above-mentioned invention, the recognition processing unit can be configured as an information processing system that performs the product identification process on the product area detected by the object detection processing unit, and / or performs the price reading process on the price display area detected by the object detection processing unit.

[0025] As in the present invention, rather than performing product identification processing / price display area processing on the entire image information, by performing product identification processing / price display area processing on the product area / price display area detected by object detection processing, the processing load can be reduced and processing can be speeded up.

[0026] In the above-mentioned invention, the information processing system has an upload processing unit that uploads part or all of the pre-correction image information and correction parameters for correction for the image information selected by the image selection processing unit, and a correction reproduction processing unit that reproduces the correction using part or all of the uploaded pre-correction image information and the correction parameters, and the recognition processing unit can be configured as an information processing system that performs product identification processing and / or price reading processing on part or all of the corrected image information reproduced by the correction reproduction processing unit.

[0027] Because the product identification process / price reading process requires a high processing load, it is preferable to perform the process on a server with high processing power rather than on a portable communication terminal with low processing power, such as a smartphone. In this case, the image information to be subjected to the product identification process / price reading process is uploaded from the portable communication terminal to the server. However, when uploading the corrected image information, blank areas are included, increasing the amount of data. Therefore, by outputting correction parameters as in the present invention, it is possible to upload the image information before correction and the correction parameters, thereby reducing the amount of data to be uploaded.

[0028] The information processing system of the first invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes a computer to function as an image capture processing unit that captures images of display shelves and a recognition processing unit that performs product identification processing and / or price reading processing on image information, wherein the image capture processing unit performs correction processing on the captured image information until a predetermined image capture end condition is satisfied, and performs image capture or accepts an image capture operation when it detects that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to movement of a portable communication terminal.

[0029] The portable communication terminal used in the information processing system of the present invention can be realized by loading and executing the information processing program into a portable communication terminal. That is, the information processing program causes a portable communication terminal used in an information processing system that photographs display shelves and recognizes the products and / or prices displayed in the image information to function as an image capture processing unit that photographs the display shelves and an upload processing unit that uploads part or all of the photographed image information or corrected image information to perform a product identification process and / or a price reading process using the image information, in which the image capture processing unit performs a correction process on the photographed image information until a predetermined photographing end condition is satisfied, and performs a photograph or accepts a photographing operation when it detects that a reference point in the photographed image information or the corrected image information has moved to a predetermined position due to movement of the portable communication terminal. [Effects of the Invention]

[0030] By using the information processing system of the present invention, it is possible to reduce the number of image information to be processed while reducing the burden of photography. In addition, since it is possible to determine image information that is out of date at the store where the photography is being performed, it is possible to re-photograph the image on the spot, thereby preventing a decrease in work efficiency. [Brief explanation of the drawings]

[0031] [Figure 1] 1 is a block diagram schematically illustrating an example of a configuration of an information processing system according to the present invention. [Figure 2] FIG. 2 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Figure 3] 3 is a flowchart showing an example of the overall processing in the information processing system of the present invention. [Figure 4] 10 is a flowchart illustrating an example of a photographing process in the information processing system of the present invention. [Figure 5]10 is a flowchart illustrating an example of an image selection process in the information processing system of the present invention. [Figure 6] 10 is a flowchart showing an example of a shelf structure analysis process in the information processing system of the present invention. [Figure 7] 10 is a flowchart illustrating an example of a skip process in the information processing system of the present invention. [Figure 8] FIG. 10 is a diagram showing an example of object labels and their meanings. [Figure 9] FIG. 10 is a diagram showing an example of a process for determining whether or not an image is cut off in the vertical direction, and is a diagram showing an example of a case where a SpaceOverRack area is detected as the space above a shelf. [Figure 10] 10A and 10B are diagrams showing an example of a process for determining whether or not an image is cut off in the vertical direction, and are diagrams showing an example in which a Stock area is detected as a stock commodity area. [Figure 11] FIG. 10 is a diagram showing an example of the process of determining whether or not the image is cut off in the vertical direction, and is a diagram showing an example in which the RackBottom area is detected as the space under the shelf. [Figure 12] FIG. 10 is a diagram schematically illustrating a case where one display shelf is continuously photographed using a plurality of pieces of image information. [Figure 13] FIG. 10 is a diagram illustrating an image selection process. [Figure 14] FIG. 10 is a diagram schematically illustrating another example in which one display shelf is continuously photographed using a plurality of pieces of image information. [Figure 15] 10A and 10B are diagrams illustrating an example of image information before correction and image information after correction processing in a correction processing unit. [Figure 16] FIG. 10 is a diagram showing an example of a shelf structure table for image information #0. [Figure 17] 10 is an example of a table showing rules for selecting an action based on the relationship between objects. [Figure 18] FIG. 10 is a diagram showing an example of a shelf structure table for image information #5. [Figure 19] FIG. 11 is a diagram showing an example of a shelf structure table for image information #11. [Figure 20]FIG. 10 is a diagram showing an example of a shelf structure table for image information #14. [Figure 21] FIG. 10 is a block diagram illustrating an example of the configuration of a correction processing unit according to a second embodiment. [Figure 22] 10 is a flowchart illustrating an example of a correction process performed by a correction processing unit according to a second embodiment. [Figure 23] 10 is an example of input image information (image information before correction). [Figure 24] FIG. 2 is a diagram schematically illustrating the relationship between a target surface and a three-dimensional space. [Figure 25] 10 is an example of image information (corrected image information) to be output. [Figure 26] FIG. 10 is a front view of the rotated point cloud mapped onto a vertical plane. [Figure 27] FIG. 10 is a top view of the rotated point cloud mapped onto a horizontal plane. [Figure 28] FIG. 10 is a diagram schematically illustrating processing in an angle estimation processing unit. [Figure 29] FIG. 10 is a diagram schematically illustrating processing in an image transformation processing unit. [Figure 30] 10A and 10B are diagrams illustrating deviations in distance information when an object having many projections and recesses is photographed. [Figure 31] FIG. 10 is a diagram illustrating a method of randomly selecting a group of points from a matrix among the group of points on which distance information is plotted. [Figure 32] 10A and 10B are diagrams showing image information when unevenness is captured on the left and right sides of the intended subject, and the resulting abnormal values ​​of distance information. [Figure 33] FIG. 10 is a diagram schematically illustrating an example of a modified correction process in the second embodiment. [Figure 34] FIG. 11 is a diagram illustrating an example of a flowchart of a skip process according to the third embodiment. [Figure 35] FIG. 10 is a diagram illustrating a skip process according to the third embodiment. [Figure 36] FIG. 13 is a block diagram illustrating an example of the configuration of an information processing system according to a sixth embodiment. [Figure 37]FIG. 13 is an example of a conceptual diagram that schematically illustrates the process of Example 6. [Figure 38] 13 is a flowchart showing an example of the overall process in the sixth embodiment. [Figure 39] 13 is a flowchart illustrating an example of a photographing process in the sixth embodiment. [Figure 40] 10A and 10B are diagrams showing an example of image information before and after correction when photographing is performed for the first time; [Figure 41] FIG. 40(b) is a diagram showing an example of a state in which an object detection process is performed on the image information of FIG. [Figure 42] FIG. 10 is a diagram showing an example of a state in which a moving display is superimposed. [Figure 43] FIG. 10 is a diagram showing an example of a state in which the smartphone is slightly moved in the movement direction. [Figure 44] FIG. 10 is a diagram showing an example of a state in which a message is displayed instructing the smartphone to be moved to a predetermined position and then stopped. [Figure 45] 45A and 45B are diagrams showing an example of image information before and after correction when photographing is performed in the state of FIG. 44. [Figure 46] FIG. 45(b) is a diagram showing an example of a state in which an object detection process is performed on the image information of FIG. [Figure 47] FIG. 10 is a diagram showing an example of a state in which a moving display is superimposed. [Figure 48] FIG. 10 is a diagram showing an example of a state in which the smartphone is slightly moved in the movement direction. [Figure 49] FIG. 10 is a diagram showing an example of a state in which a message is displayed instructing the smartphone to be moved to a predetermined position and then stopped. [Figure 50] 50A and 50B are diagrams showing an example of image information before and after correction when photographing is performed in the state of FIG. 49. FIG. [Figure 51] FIG. 50(b) is a diagram showing an example of a state in which an object detection process is performed on the image information of FIG. [Figure 52] FIG. 10 is a diagram showing an example of a state in which a moving display is superimposed. [Figure 53]FIG. 10 is a diagram showing an example of a state in which the smartphone is slightly moved in the movement direction. [Figure 54] FIG. 10 is a diagram showing an example of a state in which a message is displayed instructing the smartphone to be moved to a predetermined position and then stopped. [Figure 55] 55A and 55B are diagrams showing an example of image information before and after correction when photographing is performed in the state of FIG. 54. [Figure 56] FIG. 55(b) is a diagram showing an example of a state in which object detection processing has been performed on the image information of FIG. DETAILED DESCRIPTION OF THE INVENTION

[0032] A block diagram of an example of the overall processing functions of an information processing system 1 of the present invention is shown in Fig. 1. The information processing system 1 preferably functions as a portable communication terminal 2 such as a smartphone equipped with the functions of an image capturing device 75.

[0033] The portable communication terminal 2 is a portable information processing device (computer) such as a smartphone or tablet computer that has the functions of the image capturing device 75. It may also be a laptop computer. As long as it has the functions necessary for the processing of the information processing system 1 of the present invention, its appearance and name are not important.

[0034] 2 schematically shows an example of the hardware configuration of the portable communication terminal 2 in the information processing system 1. The portable communication terminal 2 has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 that can input information, a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN, an imaging device 75 such as a camera, and various sensor devices 76.

[0035] If the portable communication terminal 2 is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals 2 such as tablet computers and smartphones, but are not limited to these.

[0036] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0037] The image capturing device 75 may be any device that captures image information using visible light or the like.

[0038] The sensor device 76 may include a sensor that detects the angle of view captured by the imaging device 75, a sensor that detects the pitch angle and roll angle of the portable communication terminal 2 or the imaging device 75, a distance measuring sensor that measures the distance from the imaging device 75 to the object being photographed, and the like, and it is desirable for the sensor device 76 to be equipped with some or all of these, but it is not limited to these.

[0039] The functions of the various means in the present invention are only logically distinct, and may be physically or practically the same area. The order of the processes in the various means of the present invention may be changed as appropriate. Also, some of the processes may be omitted. For example, the process of determining the viewpoint direction, which will be described later, may be omitted. In this case, the process may be performed on image information that has not undergone the process of determining the viewpoint direction.

[0040] In the information processing system 1, functions are distributed between the management server 3 and the portable communication terminal 2. The way in which these functions are distributed can be changed as appropriate.

[0041] The portable communication terminal 2 in the information processing system 1 has a photography processing unit 20, an image information storage unit 21, a correction processing unit 22, an object detection processing unit 23, an out-of-view determination processing unit 24, an image selection processing unit 25, and an upload processing unit .

[0042] The photography processing unit 20 photographs a subject to be photographed, for example, a display shelf in a store, using the photography device 75. When photography starts, the photography person moves the portable communication terminal 2 in a certain direction, and when the angle of view moves a certain distance, the photography processing unit 20 detects that the angle of view has moved a certain distance using a sensor that detects the angle of view in the sensor device 76, and then photographs the subject. This process is repeated until photography is completed.

[0043] For example, the person in charge of photography fixes the portable communication terminal 2 in a fixed position while pointing the camera 75 of the portable communication terminal 2 toward the display shelf, and changes the orientation of the camera 75 of the portable communication terminal 2 so that the camera 75 takes a picture from above the display shelf to below. The photography processing unit 20 takes a picture at the start of photography, and then, when the sensor that detects the angle of view of the sensor device 76 detects that the angle of view has moved a fixed distance, for example, 1 / 10 of the vertical distance of the image information, the photography processing unit 20 takes a picture with the camera 75. This is repeated until photography is finished.

[0044] This allows the photographing processing unit 20 to photograph a plurality of successive image information pieces from the top to the bottom of the display shelf, with the vertical positions of the angle of view shifted by 1 / 10 each, for example.

[0045] The image information captured by the image processing unit 20 is stored in the image information storage unit 21.

[0046] The image information storage unit 21 stores image information captured by the image capture processing unit 20.

[0047] The correction processing unit 22 performs correction processing on the image information captured by the image capturing processing unit 20 and stored in the image information storage unit 21 so that the image information is positioned in a normal facing position. The correction processing unit 22 may perform any processing as long as it can correct the image information so that the image information is positioned in a normal facing position. For example, it may perform a known keystone correction processing.

[0048] When photographing a display shelf in a store using the photographing processing unit 20, it may be difficult to ensure a sufficient distance between the display shelf to be photographed and the photographing device 75 due to circumstances such as other display shelves being located behind the photographer and narrow aisles. Furthermore, since the display shelf is located above or below the photographer's line of sight, photographing is often performed at an elevation or depression angle. As a result, the photograph cannot be taken from a direct facing position. The correction processing unit 22 corrects the photographed image information so that it appears as if the photograph was taken from a direct facing position.

[0049] The correction processing unit 22 may generate image information by performing correction processing on the image information captured by the image capturing processing unit 20, or may output parameters (correction parameters) for transforming the image information so that it is positioned in a frontal view. For example, the correction processing unit 22 may output parameters (parameters (correction parameters) used for projective transformation) for generating image information corrected by projective transformation of the image information before correction so that it is positioned in a frontal view.

[0050] The object detection processing unit 23 detects the area and type (label) of an object (target object) that appears in the image information corrected by the correction processing unit 22. The process of detecting objects from image information can be performed by using deep learning on the image information to detect objects that appear therein. In this case, the image information corrected by the correction processing unit 22 (the image information after correction) is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the area and type of the object that appears in the image information are detected. The learning model can be one in which various image information of a display shelf is provided with objects and their labels as ground truth data. An example of object labels and their meanings is shown in Figure 8.

[0051] In addition to using deep learning for object detection processing in the object detection processing unit 23, sample data of various objects (image data of the object taken from one or more directions or data such as features based thereon) can be compared with image information corrected by the correction processing unit 22, and objects can be detected using the similarity.

[0052] The object detection process is not limited to these.

[0053] The cut-off determination processing unit 24 determines whether or not there is cut-off image information in the image information captured by the image capturing processing unit 20. The cut-off determination processing includes a process for determining cut-off in the vertical direction and a process for determining cut-off in the horizontal direction.

[0054] The process of determining whether the image information showing the top or bottom of the image information photographed by the photographing processing unit 20 is cut off in the vertical direction is to determine whether a predetermined object has been detected in the object detection processing by the object detection processing unit 23, for example, whether the space above the shelf (SpaceOverRack in the label in FIG. 8) or the stock product area (Stock in the label in FIG. 8) and the area below the shelf (RackBottom in the label in FIG. 8) are included. If the predetermined object is not detected, it is determined that there is image information that is cut off in the vertical direction.

[0055] An example of the process for determining whether or not the image is cut off in the vertical direction is shown in Figures 9 to 11. Figure 9 shows a case where a "SpaceOverRack" area is detected as the space above the shelf, Figure 10 shows a case where a "Stock" area is detected as the stock product area, and Figure 11 shows a case where a "RackBottom" area is detected as the space below the shelf.

[0056] The process of determining whether or not a part of the image is cut off in the horizontal direction is performed based on the positional relationship of a predetermined object in the image information captured by the image capture processor 20, among the objects detected by the object detection processor 23 during object detection. For example, this can be determined based on whether a stock product area (Stock in the label in FIG. 8), a shelf product area (Faces in the label in FIG. 8), a price display area for hanging products (HangingShelf in the label in FIG. 8), a hanging product area (HangingFaces in the label in FIG. 8), etc. are detected at a certain size or larger in the horizontal center of the image information (however, these objects that exist within a certain range from the right or left edge are excluded from the determination as being on an adjacent shelf), or whether a gap between the left and right sides of a shelf (border in the label in FIG. 8), a vertical structural member of the shelf (pillar in the label in FIG. 8), etc. are within a certain range from the left or right edge. If a predetermined object is not detected at a certain size or larger in the horizontal center, or if the vertical structural member of the shelf is not within a certain range from the left or right edge, it is determined that there is image information that is cut off in the horizontal direction.

[0057] When the cut-off determination processing unit 24 determines that there is cut-off image information, a predetermined warning such as that the image was not photographed properly or that the image was photographed in a cut-off state is displayed on the display device 72 of the portable communication terminal 2, and the user is prompted to photograph again.

[0058] The image selection processor 25 performs image selection processing to select image information from the image information captured by the image capture processor 20, which is then uploaded to the management server 3 for product identification and / or price reading. That is, when the image capture processor 20 continuously captures image information at a fixed amount of movement, such as a movement in the angle of view, as in the present invention, 10 or more images may be captured when capturing an image of a single display shelf. This is schematically shown in FIG. 12. FIG. 12 shows a case in which 14 images of a single display shelf are continuously captured, with the capture range for each image indicated in parentheses. The amount of movement is preferably the amount of movement in the angle of view, but is not limited to this. For example, it may be the amount of movement of the portable communication terminal 2.

[0059] 12, performing product and / or price tag identification processing on all image information captured of one display shelf requires a large processing time and processing load, and is wasteful of processing. Therefore, from the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21, image information to be uploaded to the management server 3 and subjected to product and / or price tag identification processing is selected.

[0060] The image selection process in the image selection processing unit 25 analyzes the shelf structure of the image information captured from above or below the display shelf to be processed, and selects the first image information to be processed from a predetermined position in the image information, preferably the bottom or top end, as the tracking object. Then, the image selection processing unit 25 skips to the image information that contains the tracking object within its angle of view and is two or more images away from the image information (not adjacent to the image information), preferably the furthest away, and selects the image information to be skipped as the image information to be processed, and performs the same process as described above. This process is repeated until the final image information is reached. Figure 13 shows a schematic diagram of the image selection process.

[0061] For example, as shown in Fig. 14, suppose that the photography processing unit 20 takes 18 consecutive images (image information (1) to (18)) of a single display shelf from bottom to top at each movement amount of a predetermined angle of view. The correction processing unit 22 performs correction processing on each of these image information, the object detection processing unit 23 performs object detection processing, and the cut-out determination processing unit 24 performs cut-out determination processing. Then, the image selection processing unit 25 uses the tracking object to select each of the image information (1), (5), (11), and (18) as the image information to be processed.

[0062] The upload processing unit 26 sends the image information (image information before correction) selected by the image selection processing unit 25 and the correction parameters for performing the correction on that image information output by the correction processing unit 22 in the correction processing to the management server 3. Furthermore, when sending the corrected image information to the management server 3, it is not necessary to send the correction parameters for performing the correction processing. For example, in the case of FIG. 14, the image information before correction and the correction parameters for each of the image information (1), (5), (11), and (18) are sent to the management server 3.

[0063] The management server 3 includes an image information reception processing unit 30 , a correction reproduction processing unit 31 , and a recognition processing unit 32 .

[0064] The image information reception processing unit 30 receives image information uploaded from the upload processing unit 26 of the portable communication terminal 2 and correction parameters for performing the correction on the image information output by the correction processing unit 22 in the correction process. For example, in the case of Fig. 14, the image information reception processing unit 30 receives the pre-correction image information and correction parameters for each of the image information (1), (5), (11), and (18). When the upload processing unit 26 uploads corrected image information, the correction parameters for performing the correction output by the correction process are not required.

[0065] The correction reproduction processing unit 31 executes correction processing on the image information before correction using the parameters when the image information reception processing unit 30 receives the image information before correction and parameters for performing correction. When the image information corrected by the image information reception processing unit 30 is received from a portable terminal, the correction reproduction processing unit 31 is not necessary.

[0066] The recognition processing unit 32 performs a commodity identification process and / or a price reading process on the corrected image information. The recognition processing unit 32 includes a commodity identification processing unit 320 and a price reading processing unit 321.

[0067] The product identification processing unit 320 executes a process for identifying a product in the corrected image information. For example, the product may be identified using deep learning, or may be identified using image matching processing. Various known techniques can be used as the product identification processing unit 320.

[0068] When identifying products using deep learning, it is possible to first identify the area where the products are located (face area), and then perform a process to identify which products are displayed in that face area. The face area is an area that is the outline of the product or a predetermined area that includes the product, preferably a rectangular area.

[0069] In this case, the corrected image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the area (face area) where the product is located in the image information is identified. The learning model can be one in which various image information of display shelves is provided with correct data for the area where the product is located (face area). Furthermore, when performing product identification processing using deep learning, areas in the corrected image information labeled "Faces" (shelf-mounted product area) or "HangingFaces" (hanging product area) by the object detection processing unit 23 (areas determined to contain products) can be input as processing targets, and face area identification processing using deep learning can be performed on those areas. Note that the shelf-mounted product area and hanging product area are simply referred to as product areas.

[0070] Next, the image information of the face area is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the product identification information shown in the image information of the face area is identified. The learning model can be one in which image information of various face areas is given correct answer data with product identification information (product identification information) as a label. The product identification information can be any information that can identify the product, such as the product name or JAN code.

[0071] Alternatively, the identification of the face area and the identification of the product identification information may be performed as a single process. In this case, for example, the following process can be executed. In this case, the corrected image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the face area captured in the image information and the identification information of the product in the face area are identified. The learning model can be various images of display shelves, to which corrective data is added, labeled with the face area and product identification information. The product identification information can be any information that can identify the product, such as the product name or JAN code. In addition, when performing product identification processing using deep learning, the object detection processing unit 23 can input areas (product areas) labeled with "Faces" or "Hanging Faces" in the corrected image information as processing targets, and perform processing to identify the face area and product identification information for those areas using deep learning.

[0072] Furthermore, when image matching processing is used, sample data showing the appearance of various products (image information showing the appearance of the product from one or more directions or data such as feature quantities based thereon) may be compared with the corrected image information, and the product may be identified using the degree of similarity. When image matching processing is performed, the object detection processing unit 23 may perform image matching processing on areas (product areas) labeled with "Faces" or "Hanging Faces" in the corrected image information.

[0073] The price reading processor 321 executes a process of reading the price from the price tag in the corrected image information. For example, the price may be read using deep learning, character recognition processing, or image matching processing. Various known technologies can be used for the price reading processor 321.

[0074] When reading prices using deep learning, for example, the following processing can be performed. In this case, the corrected image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the price tag area captured in the image information is identified. The learning model can be one in which various images of display shelves are provided with corrective data for the price tag area. Furthermore, when performing price tag identification processing using deep learning, the object detection processing unit 23 can input areas in the corrected image information labeled "Shelf" (shelf area with price display) or "HangingShelf" (hanging shelf price display area) as the processing target (hereinafter, areas with price display, such as shelf area with price display and hanging shelf price display area, are simply referred to as price display areas), and perform price tag identification processing using deep learning on those areas.

[0075] Furthermore, when image matching processing is used, sample data showing the appearance of various price tags (image information showing the appearance of a price tag from one or more directions or data such as features based thereon) may be compared with the corrected image information, and the price tag area may be identified using the degree of similarity.When image matching processing is performed, object detection processing unit 23 may perform image matching processing on areas in the corrected image information labeled "Shelf" or "HangingShelf" (price display areas determined to contain price tags).

[0076] Character recognition processing is performed on the area of ​​the price tag identified using deep learning, image matching processing, etc., and the price is read.

[0077] The price reading process of the price reading processor 321 may involve identifying the area of ​​a price tag and then performing character recognition processing on that area, or it may involve inputting corrected image information and reading the price of the price tag shown in that image information. A learning model may be one in which correct data is provided for the price in the area of ​​a price tag in various image information photographs of a display shelf. When performing price reading processing using deep learning, the object detection processor 23 may input an area in the corrected image information labeled "Shelf" or "Hanging Shelf" (a price display area determined to contain a price tag) as the processing target, and perform price reading processing using deep learning on that area. [Example]

[0078] Next, an example of processing using the information processing system 1 of the present invention will be described with reference to the flowcharts of Figures 3 to 7. In this example, a case will be described in which the information processing system 1 of the present invention is used to identify products and read their prices from image information of store shelves photographed with a camera (photographing device 75) built into a smartphone (portable communication terminal 2).

[0079] First, a person in charge of photographing the store's display shelves performs a predetermined operation using the camera of the smartphone he or she owns to photograph the store's display shelves (the object to be photographed), and the photographing processing unit 20 then performs the photographing process (S100).

[0080] The photographer performs a predetermined operation to activate predetermined application software stored in the smartphone. This activation causes the photography processing unit 20 to activate the smartphone's camera. The photographer then points the photography device 75 of the portable communication terminal 2 toward the display shelves, holds the portable communication terminal 2 in a fixed position so as to keep the photography position as constant as possible, and begins photographing the display shelves from above to below, or from below to above (S200). That is, when the photography processing unit 20 detects that a photography start operation has been performed so as to capture the top of the display shelves (the space above the display shelves or the space where stock items are placed) or the bottom (the floor on which the display shelves are placed), it takes the first photograph. The photographer then moves the portable communication terminal 2 in a fixed direction (upward or downward) to take a photograph (S210). As the shooting direction of the portable communication terminal 2 is changed, the shooting processing unit 20 automatically starts shooting (S230) when the sensor that detects the angle of view in the sensor device 76 detects that the angle of view has moved a certain distance, for example, 1 / 10 (S220). The shooting processing unit 20 repeats this process until a predetermined operation, for example, an operation to end shooting, is received (S240). This allows continuous image information of the display shelf to be shot upward or downward from the fixed position of the portable communication terminal 2.

[0081] The person in charge of photography should move the smartphone from top to bottom or bottom to top while taking the photograph so that the photographing position does not change as much as possible. In addition, for consecutively photographed image information, it is preferable that identification information be assigned sequentially from the first photographed image information to the last photographed image information so that the order in which each image was photographed can be identified.

[0082] The photography processing unit 20 may also perform the following display processing to display instruction information, such as the orientation of the camera, to the photographer. For example, the photography processing unit 20 of the portable communication terminal 2 displays the movement distance in the next angle of view as the length of an arrow on the display device 72 of the portable communication terminal 2. When the photographer changes the photography orientation of the portable communication terminal 2 in the direction of the arrow, the photography processing unit 20 calculates the movement distance in the angle of view due to the change in photography orientation and changes the displayed length of the arrow so that the arrow becomes shorter. When the length of the arrow reaches 0, the photography processing unit 20 stops displaying the arrow or displays a message such as "photography ended" on the display device 72, allowing the photographer to stop changing the photography orientation of the portable communication terminal 2. The photography processing unit 20 ends the photography processing when it detects that the portable communication terminal 2 has stopped. In this case, the entire display shelf, from top to bottom or bottom to top, can be photographed in one or more images.

[0083] By performing the above-described photographing process by the photographing processing unit 20, it is possible to photograph a plurality of consecutive image information frames, each frame shifting by a certain distance in the angle of view from above to below or from below to above the display shelf. The image information photographed by the photographing processing unit 20 is then stored in the image information storage unit 21. This is shown schematically in FIG. 12 or FIG. 14. FIG. 12 shows a case where 14 consecutive image information frames are photographed while changing the photographing direction of a single display shelf from above to below. FIG. 14 shows a case where 18 consecutive image information frames are photographed while changing the photographing direction of a single display shelf from below to above.

[0084] Then, the correction processing unit 22 executes a correction process on the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21 to correct the image information so that it is positioned in a normal position (S110). The correction process by the correction processing unit 22 may simply generate image information by correcting the image information itself, or may also output parameters for correcting the image information so that it is positioned in a normal position. If the parameters are also output, the amount of data to be uploaded during the upload process described below can be reduced by uploading the image information before correction and the parameters, compared to uploading corrected image information. For example, FIG. 15 shows an example of image information obtained by correcting the image information of (1) in FIG. 14. FIG. 15(a) shows the image information before correction of (1) in FIG. 14, and FIG. 15(b) shows the image information obtained by correcting the image information of (1) in FIG. 14 by the correction processing unit 22. The correction process by the correction processing unit 22 is preferably performed on the image information captured by the image capture processing unit 20, but may be performed on only part of the image information rather than all of it.

[0085] After the correction processing is performed by the correction processing unit 22, the object detection processing unit 23 detects the area and type of the object appearing in the image information corrected by the correction processing unit 22 (S120). That is, the image information corrected by the correction processing unit 22 is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and objects appearing in the image information are detected.

[0086] Further, the cut-off determination processing unit 24 performs cut-off determination processing on the corrected image information, and determines whether or not there is cut-off image information (whether or not there is an error) in the image information captured by the image capturing processing unit 20 (S130).

[0087] The cut-off determination processing unit 24 executes the process of determining cut-off in the vertical direction and the process of determining cut-off in the horizontal direction as described above.

[0088] As shown in Figures 9 to 11, the process of determining whether the cut-off in the vertical direction is performed by the cut-off determination processing unit 24 is to determine whether a specified object has been detected in the object detection processing by the object detection processing unit 23 in the image information captured by the shooting processing unit 20, and whether it includes, for example, the space above the shelf (SpaceOverRack in the label in Figure 8) or the stock product area (Stock in the label in Figure 8), and the area below the shelf (RackBottom in the label in Figure 8).

[0089] The process of determining whether or not the image is cut off in the left-right direction in the cut-off determination processor 24 determines the positional relationship in the image information of a predetermined object among the objects detected in the object detection process of the object detection processor 23 in the image information captured by the image capture processor 20. For example, the determination can be made based on whether a stock product area (Stock in the label in FIG. 8), a shelf product area (Faces in the label in FIG. 8), a price display area for hanging displays (HangingShelf in the label in FIG. 8), a hanging product area (HangingFaces in the label in FIG. 8), etc. are detected at a certain size or larger in the center of the left-right part of the image information (however, these objects that exist within a certain range from the right end or a certain range from the left end are excluded from the determination as being on an adjacent shelf), or whether a left-right gap between the shelf levels (border in the label in FIG. 8), a vertical structural member of the shelf (pillar in the label in FIG. 8), etc. are within a certain range from the left-right end.

[0090] As described above, when the out-of-image determination processing unit 24 determines that there is cut-off image information (there is an error) (S130), the out-of-image determination processing unit 24 displays a predetermined warning on the display device 72 of the portable communication terminal 2, such as that the image was not captured properly or that the image was cut off, and urges the user to capture the image again. On the other hand, when the out-of-image determination processing unit 24 determines that there is no cut-off image information (there is no error), the image selection processing unit 25 executes image selection processing (S140).

[0091] The image selection processing unit 25 performs image selection processing to select image information from the image information captured by the image capture processing unit 20, which is uploaded to the management server 3 and subjected to product identification processing and / or price reading processing.

[0092] The image selection processing unit 25 initializes a shelf structure table that indicates the areas of the display shelf, and sets a parameter p to an initial value of p=0 (S400). Then, the image information having identification information indicating that the image was taken first is selected as the processing object (S410). For example, if the image processing unit 20 has taken images from above to below, the image information having the image of the top, i.e., the image information having identification information of "#0" in Fig. 13, is selected as the processing object.

[0093] The image selection processing unit 25 executes a shelf structure analysis process for the image information selected as the processing target (S420). The shelf structure analysis process is a process for detecting and associating areas where products are placed on a display shelf with areas where price tags are displayed, and executes the process of the flowchart in FIG. 6, for example.

[0094] In the shelf structure analysis process, first, the image information of the parameter p is added as selected image information (S500). That is, the image information to be processed, in this case, the image information of "#0", is added as image information selected by the image selection processing unit 25.

[0095] Then, for the image information to be processed, i.e., image information "#0", the objects detected by the object detection processing unit 23 are extracted (S510). This is shown schematically in Fig. 13(a). In Fig. 13(a), "Stock", "ShelfWoPC", "Faces", and "Shelf" have been detected as objects, so these are extracted.

[0096] At this time, objects on the adjacent shelf are excluded (S520). In Fig. 13(a), there are no objects on the adjacent shelf, so the process is carried out as is.

[0097] The objects are then sorted in order from the direction in which they were photographed (S530). In FIG. 13, the photographs were taken from top to bottom, so the objects are sorted from top to bottom. In the case of FIG. 13(a), the objects are sorted in the order of "Stock," "ShelfWoPC," "Faces," and "Shelf." Each sorted object is then written to the shelf structure table (S540). FIG. 16 shows an example of a shelf structure table for the image information to be processed (image information #0).

[0098] Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560). For example, the image selection processing unit 25 selects an action for the object based on the rule, an example of which is shown in Fig. 17, and executes the process of creating pairs.

[0099] Figure 17(a) is a table showing names of actions to be performed based on the relationship between the current object and the object immediately below, and Figure 17(b) is a table associating action names with action content. The rules in Figure 17 are merely examples and are not limiting. It is preferable that the rules associate the relationship between adjacent objects and action names with action content, but any rule may be used.

[0100] In the case of the shelf structure table of FIG. 16, first, the topmost object is "Stock" and the object immediately below it is "ShelfWoPC," so the table of FIG. 17 is referenced and an action with the action name "pairing" is performed, i.e., these two objects are paired. Then, the next object is "ShelfWoPC" and the object immediately below it is "Faces," so the table of FIG. 17 is referenced and an action with the action name "OK" is performed, i.e., nothing is done. Then, the next object is "Faces" and the object immediately below it is "Shelf," so the table of FIG. 17 is referenced and an action with the action name "pairing" is performed, i.e., these two objects are paired. The next object is "Faces," but there is no object immediately below it, so it is determined that the action selection process for all objects in the image information to be processed has ended (S550), and the shelf structure analysis process is terminated.

[0101] When the shelf structure analysis process ends, if the image information for which the shelf structure analysis process has been performed is not the last image information (S430), the image selection processing unit 25 performs skip processing to select a skip destination image (S440).

[0102] The skip process is a process of finding other image information that includes a tracking object included in the image information to be processed, i.e., a process of finding image information to be skipped to. For example, the object at the bottom of the image information to be processed is set as the tracking object, and other image information that includes the area of ​​the tracking object at its top is found, and the found image information is set as the skip destination image information to be processed next.

[0103] In the skip process, a tracking object is determined in the image information (image information #0) to be processed by parameter p, and its area is recorded (S600). For example, the bottommost object in image information #0, the "Shelf" object, is set as the tracking object, and its area is recorded.

[0104] The parameter p is incremented to p=1 (S610). Since the image information is not yet the last (S620), the image information with the next identification information after the image information with identification information #0, i.e., the image information with identification information #1, is selected, and the area of ​​the image information recorded in S600 is determined (S630). At this time, an image matching process may be performed between the image information of the tracking object area recorded in S600 and the image information of #1 to determine whether there is an area similar to the tracking object area. Furthermore, among the objects in the image information of #1, the area of ​​an object of the same type as the tracking object, "Shelf," may be matched. If the similarity is equal to or greater than a predetermined threshold, it is determined that the tracking object is present in the image information of #1.

[0105] If the image information of the incremented p has an area corresponding to the tracking object (S640), p is incremented to p=2 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650). "There is an area corresponding to the tracking object" means that there is an area in the image information whose similarity to the tracking object area is equal to or greater than a predetermined threshold, and "there is no area corresponding to the tracking object" means that there is no area in the image information whose similarity to the tracking object area is equal to or greater than a predetermined threshold, for example, when there is no area corresponding to the tracking object, or when the area corresponding to the tracking object is only partial (cut off).

[0106] By repeating the above process, the image information to be skipped is searched for.

[0107] For example, when p=6 (image information #6), if there is no area corresponding to the tracking object (if there is no area with a similarity greater than or equal to a predetermined threshold), p is decremented, p=5 is set, and the skip process is terminated (S650).

[0108] When the skip process is completed (S440), the image information of p=5, that is, the image information of #5, is selected as the processing target (S410), as shown in Figure 13(b).

[0109] Then, a shelf structure analysis process is performed on the image information of p=5, the image information of #5 (S420), and the image information of #5 is added as selected image information (S500). Then, the process from S510 onwards is performed on the image information of #5 in the same manner as described above, extracting objects from the image information of #5 (S510), excluding objects on adjacent shelves (S520), and sorting them in order (S530). The objects are then written into the shelf structure table. An example of the shelf structure table for the image information of #5 is shown in FIG. 18. Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0110] After the shelf structure analysis process is performed on the image information of #5, the skip process is performed on the image information of #0 (S440).

[0111] The image selection processing unit 25 records the area of ​​the bottommost "Shelf" object as the tracking object in the image information of #5 (S600). Then, p is incremented to p=7 (S610). Because p≦14, this is not the last image information (S620), so image matching processing is performed between the image information of the tracking object area recorded in S600 (image information of the bottommost "Shelf" area in the image information of #5) and the image information of the object "Shelf" area in the image information of #7, and it is determined whether there is an area similar to the tracking object area (S630).

[0112] If the incremented image information of p=7 (image information of #7) contains an area corresponding to the tracking object (S640), p is incremented to p=8 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650).

[0113] By repeating the above process, the image information to be skipped is searched for.

[0114] For example, when p=12 (image information #12), if there is no area corresponding to the tracking object (if there is no area with a similarity greater than or equal to a predetermined threshold), p is decremented, p=11 is set, and the skip process is terminated (S650).

[0115] When the skip process is completed (S440), the image information of p=11, that is, the image information of #11, is selected as the processing target (S410), as shown in Figure 13(c).

[0116] Then, a shelf structure analysis process is performed on the image information of p=11 and the image information of #11 (S420), and the image information of #11 is added as selected image information (S500). Then, the process from S510 onwards is performed on the image information of #11 in the same manner as described above, and objects are extracted from the image information of #11 (S510), objects on adjacent shelves are excluded (S520), and the images are sorted in order (S530). The objects are then written into the shelf structure table. An example of the shelf structure table for the image information of #11 is shown in FIG. 19. Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0117] After the shelf structure analysis process is performed on the image information of #11, the skip process is performed on the image information of #5 (S440).

[0118] The image selection processing unit 25 records the area of ​​the bottommost "Faces" object as the tracking object in the image information of #11 (S600). Then, p is incremented to p=12 (S610). Because p≦14, this is not the last image information (S620), so image matching processing is performed between the image information of the tracking object area recorded in S600 (image information of the bottommost "Faces" area in the image information of #11) and the image information of the object "Faces" area in the image information of #12, and it is determined whether there is an area similar to the tracking object area (S630).

[0119] If the incremented image information of p=12 (image information of #12) contains an area corresponding to the tracking object (S640), p is incremented to p=13 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650).

[0120] By repeating the above process, the image information to be skipped is searched for.

[0121] For example, when p=14, there is an area corresponding to the tracking object up to (image information #14), and if p is incremented to p=15 (S610), p≦14 will be obtained, so p is considered to have passed the last image information (S620), p is decremented to p=14, and the skip process is terminated (S650).

[0122] When the skip process is completed (S440), the image information of p=14, that is, the image information of #14, is selected as the processing target (S410), as shown in Figure 13(d).

[0123] Then, a shelf structure analysis process is performed on the image information of p=14 and the image information of #14 (S420), and the image information of #14 is added as selected image information (S500). Then, the process from S510 onwards is performed on the image information of #14 in the same manner as described above, extracting objects from the image information of #14 (S510), excluding objects on adjacent shelves (S520), and sorting them in order (S530). The objects are then written into the shelf structure table. An example of the shelf structure table for the image information of #14 is shown in FIG. 20. Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0124] When the shelf structure analysis process is executed for the image information of #14, since it is the last image information (S430), the image selection processing section 25 ends the image selection process.

[0125] By performing the above-described processing, the image selection processing unit 25 can select each of the image information #0, #5, #11, and #14 in FIG. 12 as image information to be uploaded.

[0126] After the image selection process is completed, upload processing unit 26 sends the image information (image information before correction) selected by image selection processing unit 25 and the parameters for performing correction on that image information output by correction processing unit 22 in the correction process to management server 3 (S150). That is, upload processing unit 26 sends the image information before correction for #0, #5, #11, and #14 and the parameters for performing correction to management server 3.

[0127] The upload processing unit 26 may send the corrected image information of #0, #5, #11, and #14 to the management server 3.

[0128] In the upload process, the upload processing unit 26 may send a shelf structure table corresponding to each piece of image information.

[0129] Then, when the image information reception processing unit 30 of the management server 3 receives each piece of pre-correction image information and the parameters for performing the correction uploaded from the portable communication terminal 2, i.e., each piece of pre-correction image information #0, #5, #11, and #14 and the parameters for performing the correction, the correction reproduction processing unit 31 performs a correction reproduction process on the uploaded pre-correction image information using the parameters for performing the correction process (S160). Therefore, each piece of pre-correction image information #0, #5, #11, and #14 is transformed into corrected image information using the parameters for performing the correction. The correction reproduction process is similar to the correction process in the portable communication terminal 2, but since it simply transforms (projectively transforms) the pre-correction image information using the parameters, it can be performed by calculation processing, allowing for high-speed processing.

[0130] It should be noted that if the image information reception processing unit 30 receives the corrected image information, the correction reproduction processing in the correction reproduction processing unit 31 is not necessary.

[0131] Then, the product identification processing unit 320 in the management server 3 performs product identification processing on each corrected image information, and identifies the products displayed in the areas where products are displayed in each image information (areas labeled "Faces" or "Hanging Faces" (areas where it is determined that products are present)) (S170, S180).

[0132] That is, a commodity identification process is executed for each of the corrected image information #0, #5, #11, and #14, and the commodity displayed in the commodity display area in each of the image information is identified (S170, S180).

[0133] Similarly, the price reading processing unit 321 in the management server 3 performs a price reading process on each corrected image information, and reads the price of each price tag in the area where the price tag is located in each image information (the area labeled "Shelf" or "HangingShelf" (the price display area where it is determined that a price tag is located)) (S175, S185).

[0134] By performing the above-described processing, it is possible to take appropriate photographs when photographing in a store, and to identify products displayed on display shelves.

[0135] If the shelf structure table corresponding to each image information is also uploaded during the upload process, the structure of a single display shelf can be identified by combining the shelf structure tables based on the tracking object. Therefore, based on the results of the product identification process and price reading process for each image information, it is also possible to identify which product is displayed on which shelf and how much the price tag says on it. [Example]

[0136] In the above-described first embodiment, various known correction processing techniques capable of executing correction processing to achieve a head-on position have been described as the correction processing in the correction processing unit 22. However, a case where correction processing with a reduced processing load can be realized can be described. This allows correction processing to be executed with a reduced load even in a portable communication terminal 2 with low processing power, such as a smartphone.

[0137] An example of the configuration of the correction processing unit 22 in this embodiment is shown in FIG. 21, and an example of the correction processing in the correction processing unit 22 is shown in the flowchart of FIG.

[0138] In this embodiment, the correction processing unit 22 regards the front surface of the object to be photographed as a vertical rectangle (including a square) in three-dimensional space (this surface is called the object surface), and performs correction processing by estimating the positional relationship between the portable communication terminal 2 and the object surface in three-dimensional space using sensor information detected by the sensor device 76 of the portable communication terminal 2. Fig. 23 shows an example of image information (image information before correction) photographed by the photographing device 75.

[0139] The correction processing unit 22 includes a sensor information input reception processing unit 220 , an angle estimation processing unit 221 , an image transformation processing unit 222 , and an output processing unit 223 .

[0140] The sensor information input reception processing unit 220 receives input of information from the sensor device 76, which detects the pitch angle and roll angle of the portable communication terminal 2 or the image capturing device 75. It also receives input of distance information between the image capturing device 75 and the image capturing object from a sensor (ranging sensor) that measures the distance to the image capturing object. The distance information obtained is distance information from the image capturing device 75 to each point on the image capturing object.

[0141] The angle estimation processing unit 221 converts the distance information to each point on the object to be photographed, which is received by the sensor information input reception processing unit 220, into three-dimensional information, and rotates this point cloud based on the roll angle and pitch angle. Because the object plane is vertical, the result of the rotation is a plane that is roughly perpendicular to the xz plane, as shown in Figure 24. Then, by calculating a regression line of these points on the xz plane, the perpendicular line drawn from the image capture device 75 to this line is estimated as the yaw angle.

[0142] The image transformation processing unit 222 uses the yaw angle estimated by the angle estimation processing unit 221 to transform the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21 so that the image information is positioned directly facing the image capture device 75. The target rectangular shape of the target surface is determined by assuming four vectors with the image capture device 75 as endpoints, which represent the angle of view of the image capture device 75 in three-dimensional space. The vectors are then rotated based on the roll angle and pitch angle input received by the sensor information input reception processing unit 220, and the angle estimation processing unit 221 calculates four intersections with a vertical plane tilted to the left or right by the yaw angle estimated by the angle estimation processing unit 221. The image information captured by the image capture processing unit 20 is then subjected to a projective transformation process using OpenCV or the like using the calculated four intersections, thereby performing image transformation processing and correcting the image information from a directly facing position. FIG. 25 shows an example of image information corrected from the image information of FIG. 23.

[0143] The output processing unit 223 outputs the image information corrected by the correction processing unit 22. For example, the output processing unit 223 outputs the corrected image information as shown in Fig. 25. The output processing unit 223 may also output parameters for correction by the correction processing unit 22, i.e., parameters for performing projective transformation processing.

[0144] Next, the correction processing of the correction processing unit 22 in this second embodiment will be described with reference to the flowchart of FIG.

[0145] First, a photographer photographs the store's display shelves (the object to be photographed) using the camera of the smartphone he or she carries. The photography processing unit 20 stores the image information photographed by the smartphone camera in the image information storage unit 21 (S700). An example of this image information is shown in FIG. 23. The image information in FIG. 23 was photographed from a diagonally downward direction of the display shelves, looking upward (at an elevation angle), and the display shelves are slightly rotated to the left (the pitch angle is 18.8 degrees upward, and the roll angle is 4.6 degrees counterclockwise).

[0146] In addition, the sensor information input reception processing unit 220 receives input of information on the pitch angle and roll angle of the smartphone or camera when capturing image information with the imaging device 75, as measured by the sensor device 76, and information on the distance to each point on the object being photographed (S700).

[0147] Distance information may be acquired from any number of locations, for example, from several tens to a hundred locations, for the object being photographed. Note that the number of locations from which distance information is acquired may be any number as long as sufficient accuracy can be obtained.

[0148] The angle estimation processing unit 221 plots the distance information received as input by the sensor information input reception processing unit 220 in a three-dimensional space and converts the distance information into a point cloud in the three-dimensional space (S710). Then, the angle estimation processing unit 221 rotates this point cloud using the pitch angle and roll angle information received as input by the sensor information input reception processing unit 220 (S720).

[0149] To generate corrected image information (trapezoid-corrected image information) such that the products displayed on the shelves on the target surface are upright and the shelves are horizontal, i.e., to correct the target surface of the image information captured by the image capture processing unit 20 so that it faces directly at a certain position, it is necessary to calculate the degree to which the target surface is tilted from the orientation facing the camera. Therefore, the point cloud after rotation in S720 is mapped onto a horizontal plane (xz plane), and a regression line is calculated using a robust estimation algorithm such as RANSAC (S730). Figure 26 is a front view (xy plane in Figure 24) of the point cloud after rotation in S720 mapped onto a vertical plane, and Figure 27 is a top view (xz plane in Figure 24) of the point cloud after rotation in S720 mapped onto a horizontal plane. Figure 28 also shows a schematic diagram of the processing in the angle estimation processing unit 221.

[0150] Then, the angle (yaw angle) of the target surface relative to the camera's optical axis can be calculated from this regression line. Specifically, if the slope of the regression line of the point cloud is a, then the slope a of the regression line is converted to a yaw angle by calculating Equation 1 (S740). In other words, the arctangent is calculated using the slope a as an argument. (Number 1) Yaw angle = atan(a)

[0151] Since the angle estimation processing unit 221 can estimate the yaw angle through the above processing, the image transformation processing unit 222 uses the estimated yaw angle to transform the image information captured by the photography processing unit 20 and stored in the image information storage unit 21 so that the image is positioned directly facing the photography device 75. This processing is schematically shown in Fig. 29.

[0152] Since the angle estimation processing unit 221 has estimated the yaw angle and the positional relationship between the camera and the target surface in three-dimensional space has been determined, the image transformation processing unit 222 sets four lines (vectors) in three-dimensional space connecting the four vertices formed by the horizontal plane, the camera position, and the camera's angle of view (S750).Then, these four lines (vectors) are rotated using the roll angle and pitch angle accepted as input by the sensor information input acceptance processing unit 220 and the yaw angle estimated by the angle estimation processing unit 221 (S760).

[0153] The image transformation processing unit 222 calculates a quadrangle of a cross section obtained by cutting the four straight lines rotated in S760 with the target plane (S770). The image transformation processing unit 222 performs projective transformation on the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21 with respect to this target quadrangle to generate corrected image information (S780). The image information obtained by performing the correction processing on the image information in Figure 23 by the correction processing unit 22 is the image information in Figure 25.

[0154] By executing the above processing, the processing of the correction processing unit 22 can be executed.

[0155] Then, the output processing unit 223 outputs the image information (for example, FIG. 25) corrected by the correction processing unit 22. The output processing unit 223 may also output parameters for projective transformation of the target rectangle as parameters to be used during correction. By outputting the parameters, the upload processing unit 26 can reproduce the correction in the correction reproduction processing unit 31 of the management server 3 by sending the image information before correction and the parameters instead of the image information after correction, thereby reducing the amount of data to be uploaded.

[0156] In addition to the above-described processing by the correction processing unit 22, the following processing can also be performed.

[0157] Specifically, if the object being photographed is uneven, such as a display shelf displaying merchandise, the unevenness of the object being photographed has a much stronger correlation in the horizontal direction than in the vertical direction. Therefore, if the left-right direction of the camera 75 is slightly misaligned with the horizontal direction of the display shelf, the distance information for horizontally arranged objects is likely to exhibit a "step-like difference" in which the objects appear close up to a certain point and far away from there. If several such rows are mixed into the acquired distance information, the resulting estimation result will be tilted relative to the actual object surface. This is schematically illustrated in Figure 30. Figure 30(a) shows image information captured of an uneven display shelf, such as a display shelf displaying merchandise. Figure 30(a) shows that the position (point) from which distance information is acquired may be the position of a product located in the foreground or the position of a shelf with no products in the background. In this case, the point cloud positional relationship shown in Figure 30(b) results, which is likely to result in tilted estimation results.

[0158] Therefore, in the correction process of the second embodiment, the target surface may be further divided into matrices of a predetermined size, and distance information of points scattered pseudo-randomly in each matrix may be acquired. This makes it possible to randomly select points from the matrix among the points plotted with distance information when converting the distance information into a point cloud in three-dimensional space in S710. This is schematically shown in FIG. 31.

[0159] The above processing can improve the accuracy of the estimation result.

[0160] Furthermore, the correction processing unit 22 may perform the following processing.

[0161] Specifically, when the object being photographed is a display shelf, the photographer typically stands near the center of the display shelf and takes a photograph so that the entire left and right sides of the shelf are captured. In this case, there is a possibility that parts of adjacent shelves will appear near the left and right edges of the captured image. An example of image information illustrating this is shown in Figure 32. In this type of image information, if the left and right shelves protrude forward or are recessed backward compared to the display shelf that was originally being photographed, the regression line in S730 may be tilted. In the captured image information shown in Figure 32(a), the shelf that appears on the right side of the display shelf that was originally being photographed protrudes forward. Therefore, when the distance information is plotted in three-dimensional space, an abnormal value appears on the right side, as shown in the top view (xz plane) of Figure 32(b). If these point clouds are processed as they are, the slope of the regression line will be affected.

[0162] Therefore, in the correction processing in the correction processing unit 22 of the second embodiment, when a regression line is calculated using processing such as RANSAC, the following processing may be executed.

[0163] First, using processing such as RANSAC, a predetermined number of points are randomly selected from the rotated point cloud in S720 to find a regression line. Then, for all point clouds, the number of points whose error from the regression line falls within a threshold is used as an evaluation function, and this operation is repeated multiple times, and the regression line with the most points is selected as the final regression line. For example, when finding a regression line from a point cloud of 100 points, a regression line is found by selecting three points as the predetermined number of points, and sufficient accuracy can be achieved by repeating the process 500 times.

[0164] In addition, this evaluation function can be modified to lower the weight when selecting points on the outer edge, thereby reducing the impact of unevenness of the display shelves reflected near the left and right edges. One way to modify the weighting is to count points on the outer edge as 0.5 instead of 1.

[0165] More specifically, as shown in Figure 33, the point cloud acquisition section is divided into a predetermined number of parts in the x-axis direction, for example, four equal parts, and the weighting of the number of points in the outer sections is changed depending on the position on the outer edge.

[0166] The correction processing in the correction processing unit 22 in the second embodiment can be applied not only to image information captured at an elevation angle or depression angle, but also to image information captured from the left and right directions. [Example]

[0167] In the skip processing of the image selection processing unit 25 in the first embodiment described above, the image information to be skipped is identified by using the processing of comparing the image information in the tracking object area with similar image information.

[0168] Therefore, since comparing image information would increase the processing load, a case in which image information comparison processing is not used will be described in this Example 3. An example of processing in this case is shown in the flowchart of Fig. 34. Note that the process up to the skip processing in the image selection processing unit 25 is the same as in Example 1 or Example 2.

[0169] A tracking object is determined in the image information (image information #0) to be processed by parameter p, and the position information (coordinate information) of the rectangle that constitutes that area is recorded (S800). For example, the bottommost object in image information #0, the "Shelf" object, is determined as the tracking object, and the position information of the rectangle that constitutes that area is recorded. The position of the tracking object has been identified by object detection processing in object detection processing unit 23.

[0170] The parameter p is incremented to p=1 (S810). Since the image information is not yet the last (S820), the image information with the identification information next to the image information with the identification information #0, i.e., the image information with the identification information #1, is selected, and the amount of movement (movement distance) of the angle of view from the image information with the identification information #0 to the image information with the identification information #1 is determined (S830). Note that the amount of movement of the angle of view is the amount of movement by which photography is automatically performed in the photography processing unit 20. For example, the amount of movement is 1 / 10 of the angle of view.

[0171] Then, the coordinate value corresponding to the amount of movement in S830 is added to the position information of the rectangle that constitutes the area of ​​the tracking object recorded in S800 (S840). This makes it possible to identify where the tracking object in the image information of #0 is located in the image information of #1.

[0172] If the rectangular area that constitutes the tracking object is included in the image information in #1, it is determined that the tracking object is present in the image information in #1.

[0173] If the image information of the incremented p contains an area corresponding to the tracking object (S850), p is incremented to p=2 (S810). On the other hand, if there is no area corresponding to the tracking object (S850), p is decremented (S860).

[0174] By repeating the above process, the image information to be skipped is searched for.

[0175] FIG. 35 schematically shows the process of determining whether the tracking object is included in the image information indicated by the parameter p. FIG. 35(a) is a diagram schematically showing the amount of movement in the angle of view of the image information for p=n and the image information for p=n+1. One movement results in an amount of movement in the angle of view of (α, β). Therefore, the position information (x, y) of a rectangle that constitutes the area of ​​the tracking object in the image information for p=n becomes the position information (x+α, y+β) of a rectangle that constitutes the area of ​​the corresponding tracking object in the image information for p=n+1. FIG. 35(b) shows this diagrammatically. Therefore, it is sufficient that the position coordinates (x+α, y+β) of the vertices of the rectangle in the image information for p=n+1 are included in the area of ​​the image information for p=n+1. Although only one point is shown here, it is possible to determine whether the tracking object is included in the image information of p by determining that the four points that make up a rectangle, or the two points that make up the bottom edge (when moving from top to bottom) or the two points that make up the top edge (when moving from bottom to top) are included.

[0176] By performing the above process, it is possible to identify the image information to be skipped without performing a process of comparing the image information in the tracking object area with similar image information, thereby reducing the processing load on the image selection processing unit 25.

[0177] In addition, if the corrected image information is not used in the skip processing, the shape and position information of the tracking object and the position information of the image information of the parameter p may be calculated using the yaw angle identified by the correction processing unit 22 and the roll angle and pitch angle detected by the sensor device 76, and then processing may be performed. [Example]

[0178] As a variation of Examples 1 to 3, the upload processing unit 26 may be configured to cut out and upload image information of areas in the corrected image information where it is determined that there is a product (areas labeled "Faces" or "Hanging Faces" by the object detection processing unit 23) and image information of price display areas in the corrected image information where it is determined that there is a price tag (areas labeled "Shelf" or "Hanging Shelf" by the object detection processing unit 23), rather than uploading the corrected image information or the uncorrected image information and correction parameters.

[0179] In this case, the management server 3 does not need to be provided with the correction reproduction processing unit 31 .

[0180] The product identification processing unit 320 of the management server 3 performs a product identification process similar to that of Example 1 on the image information of the area in which the image information reception processing unit 30 has determined that a product is present in the corrected image information received from the portable communication terminal 2.

[0181] In addition, the price reading processing unit 321 of the management server 3 performs a price reading process similar to that of Example 1 on the image information of the price display area in which it is determined that a price tag is present in the corrected image information received from the portable communication terminal 2 by the image information receiving processing unit 30.

[0182] This allows the amount of image information data to be uploaded to be further reduced. [Example]

[0183] In the photography processing unit 20 of the first to fourth embodiments described above, a single display shelf is continuously photographed using a plurality of pieces of image information, but if the aisle width is sufficiently wide so that a single display shelf can be photographed using a single piece of image information, the photography processing unit 20 may photograph a single display shelf using a single piece of image information. In this case, there is no need to perform image selection processing in the image selection processing unit 25. That is, the correction processing in the correction processing unit 22, such as in the first or second embodiment, is performed on the image information photographed by the photography processing unit 20, and the object detection processing in the object detection processing unit 23 is performed on the corrected image information.

[0184] After the cut-off determination processing is performed in the cut-off determination processing unit 24, the upload processing unit 26 uploads the corrected image information or the uncorrected image information and the correction parameters. Then, in the management server 3, the correction reproduction processing unit 31 performs the correction reproduction processing, the product identification processing unit 320 performs the product identification processing, and the price reading processing unit 321 performs the price reading processing.

[0185] Furthermore, when performing the same processing as in the fourth embodiment, after the cut-out determination processing is performed in the cut-out determination processing unit 24, the image information of the area in the corrected image information where it is determined that a product is present and the image information of the price display area where it is determined that a price tag is present are uploaded by the upload processing unit 26. Then, in the management server 3, the product identification processing unit 320 performs the product identification processing, and the price reading processing unit 321 performs the price reading processing. [Example]

[0186] A modified example of the photography processing unit 20 will be described. In the photography processing unit 20 of the first embodiment, when photography starts, the photographer moves the portable communication terminal 2 in a certain direction, and when the angle of view moves a certain distance, the sensor for detecting the angle of view in the sensor device 76 detects that the angle of view has moved a certain distance and photographs the subject, and this process is repeated until the photographing is completed. In this case, if the photographer quickly moves the photographing direction of the portable communication terminal 2, blurring may occur when photographing with the photographing device 75, and the photograph may not be captured to an extent that allows for analysis of image information.

[0187] Therefore, in the photographing processing unit 20 of this embodiment, a case will be described in which, in addition to photographing processing in the photographing processing unit 20, out-of-image determination processing in the object detection processing unit 23 and out-of-image determination processing unit 24 is included. An example of a block diagram of the configuration of the information processing system 1 in this case is shown in Fig. 36.

[0188] In the photography processing unit 20 of this embodiment, after photography is performed by the photography device 75, the correction processing unit 22 performs correction processing on the photographed image information, and the object detection processing unit 23 performs object detection processing to detect objects. Then, among the objects in the image information, a marker is set on an object in the direction of movement of the portable communication terminal 2, preferably on the object at the end of the direction of movement of the portable communication terminal 2, or in its vicinity. For example, if the photographer moves the portable communication terminal 2 from top to bottom, a marker is set on an object in the downward direction, preferably on the object closest to the bottom, among the objects detected in the image information; and if the portable communication terminal 2 is moved from bottom to top, a marker is set on an object in the upward direction, preferably on the object closest to the top, among the objects detected in the image information, or in its vicinity.

[0189] The amount of movement, for example, the amount of movement of the angle of view, from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the end on the opposite side to the direction in which the portable communication terminal 2 is moved, to the marker is calculated, and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the display device 72 of the portable communication terminal 2. The direction of this arrow can be determined depending on the direction from the reference point to the marker, so it is preferable to use the direction of the arrow as the movement direction. That is, the photography processing unit 20 displays on the display device 72 an arrow or other display (movement display) indicating the direction and / or amount of movement of the portable communication terminal 2, superimposed on the image information captured by the photography device 75 of the portable communication terminal 2.

[0190] When the photographer moves the portable communication terminal 2 in the direction of the arrow, the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the distance traveled to the destination. As a result, as the photographer moves the orientation of the portable communication terminal 2, the length of the arrow becomes shorter, and when the marker reaches an arbitrary point in the original image information, for example, an arbitrary point (reference point) near the end of the image information on the opposite side from the direction in which the portable communication terminal 2 is being moved, the length of the arrow becomes zero. At this time, a movement indication is displayed indicating that the portable communication terminal 2 is being stopped, and the photography device 75 automatically takes a photograph. Note that while it is preferable for the photography device 75 to take a photograph automatically, the photography processing unit 20 may also accept the photography operation performed by the photographer.

[0191] To automatically capture images, the camera 75 captures images after it is confirmed that the person in charge of capturing images has stopped the portable communication terminal 2 and that the acceleration detected by the sensor device 76 of the portable communication terminal 2, for example, an acceleration sensor, is 0 or within a range of acceleration that can be determined as a stop. After capturing images with the camera 75, a display indicating the start of movement is displayed, for example, by calculating the amount of movement in the angle of view from the reference point to the next marker and displaying a movement such as an arrow with a length proportional to the amount of movement. This can prevent blurring and the like.

[0192] The above process is repeated until the entire display shelf has been photographed. By performing the above process, the number of images captured by the photographing processor 20 can be reduced, and the processing by the image selection processor 25 can be omitted.

[0193] FIG. 37 schematically illustrates the processing of this embodiment. FIG. 37(a) is a side view of the state in which the photographer is photographing the display shelf, and FIG. 37(b) is a front view of the state in which the photographer is photographing the display shelf. Also, p1 to p4 are the positions of the portable communication terminal 2 when photographing the display shelf with the photographing device 75 of the portable communication terminal 2. Note that the photographing positions of the portable communication terminal 2 are shown shifted in FIG. 37 for ease of understanding, but in reality, the photographing positions may be the same or shifted, with only the photographing direction of the portable communication terminal 2 changing.

[0194] The processing in this embodiment will be described using the flowcharts in Figures 38 and 39. Figure 38 is an example of a flowchart showing the overall processing in this embodiment, and Figure 39 is an example of a flowchart showing the photographing processing. The case where a smartphone is used as the portable communication terminal 2 will be described. As in the case of Figure 37, the case where the photographing direction is changed from downward to upward using the portable communication terminal 2 will be described. That is, the case where the photographing direction is changed to p1, p2, p3, and p4 will be described.

[0195] First, as in Example 1, the person in charge of photographing the store's display shelves performs a predetermined operation on the store's display shelves (the object to be photographed) using the camera of the smartphone he or she owns, and the photography processing unit 20 executes the photography process (S900).

[0196] The photographer performs a predetermined operation to activate predetermined application software stored in the smartphone. This activation causes the photography processing unit 20 to activate the smartphone's camera. The photographer then points the smartphone's camera toward the display shelves, holds the smartphone in a fixed position so as to keep the shooting position as constant as possible, and begins the photography process by moving from above to below the display shelves, or from below to above. That is, the photography processing unit 20 repeats the process from S1000 onwards until a photography end condition is met, such as capturing an image of the top of the display shelves (the space above the display shelves or the space where stock items are placed) or the bottom (the floor on which the display shelves are placed). Here, because the smartphone's shooting direction is changed from below to above, the photography end condition is when an object at the top of the display shelves (the space above the display shelves or the space where stock items are placed) is recognized.

[0197] 37, since shooting starts from below, the person in charge of shooting points the smartphone in the direction of p1, which is the shooting start position. When the shooting processing unit 20 detects that the smartphone has stopped to a certain extent using an acceleration sensor of the sensor device 75 or the like (S1000), it takes the first shot (S1010). At this time, the shooting processing unit 20 may automatically take the shot with the shooting device 75, or may issue an instruction to press the shutter of the smartphone camera and have the person in charge of shooting perform the shooting operation.

[0198] For example, Figure 40(a) shows an example of image information captured when the first image is taken so that the bottom part is visible. Figure 40(a) shows the state when the display shelf is photographed from position p1 so that the floor surface is visible.

[0199] The image information captured by the image processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 executes a correction process to correct the image to a position facing directly (S1020). This process may be a well-known keystone correction process as in the first embodiment, or a correction process as in the second embodiment. Note that if the image processing unit 20 can capture an image from a position facing directly, the correction process by the correction processing unit 22 does not need to be performed. Figure 40(b) shows the image information after the correction process.

[0200] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing unit 22 (S1030).

[0201] That is, the object (target) area and its type (label) are detected from the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing unit 22. The object detection processing in the object detection processing unit 23 can be performed using the same method as in each of the above-mentioned embodiments. Fig. 41 shows the state in which the object area and type are detected from the image information after the correction processing in Fig. 40(b). Fig. 41 shows the state in which the area and type of each object, "RackBottom", "Shelf", "Faces", and "Shelf" are detected from the bottom.

[0202] The image capture processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether a predetermined object at the top or bottom of the image capture direction, such as a "Space Over Rack" or "Rack Bottom," has been detected (S1040). Here, since "Space Over Rack," which indicates that the image capture has been completed up to the top of the display shelf, has not been detected, the image capture processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction of the movement of the smartphone's image capture direction, preferably on the object at the extreme end or in its vicinity (S1050). Since the object in question serves as a tracking object, it may also be used. In this case, since the smartphone is moved from bottom to top, a marker is set near the top of the "Shelf" object, which is located at the top. The marker and the tracking object may be the same, and all or part of the area of ​​the "Shelf" object at the top of FIG. 41 may be set as the marker.

[0203] The image processing unit 20 then calculates the amount of movement of the angle of view from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the end opposite to the direction in which the smartphone is moved, to the marker (S1060), and displays a predetermined display such as an arrow with a length proportional to the amount of movement on the smartphone display (S1070). The direction of this arrow can identify the direction in which the smartphone is moved depending on the direction from the reference point to the marker, so it is preferable to use the direction of the arrow as the movement direction. That is, the image processing unit 20 displays an arrow or other display (movement display) on the smartphone display that indicates the direction and / or amount of movement of the smartphone, superimposed on the image information captured by the smartphone camera.

[0204] FIG. 42 shows a schematic diagram of this state. At this time, the smartphone display is displaying the image information before correction (FIG. 40(b)), so the image information may be superimposed on that image information. In this case, the amount of movement in the angle of view from the reference point to the marker is calculated using the image information after correction, so it may be converted into the amount of movement in the angle of view from the reference point to the marker in the image information before correction, and the length of the arrow may be displayed. Alternatively, since the vertical deviation before and after correction is not large, even if the amount of movement is calculated based on the image information after correction, the tracking object or a marker placed near it will not deviate from the angle of view very often. Therefore, an arrow with the length of the arrow corresponding to the amount of movement calculated using the image information after correction may be superimposed on the image information before correction.

[0205] In either case, the image capturing processing unit 20 performs a moving display that indicates the direction and / or amount of movement of the smartphone, superimposed on the image displayed on the display of the smartphone.

[0206] When the photographer moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the distance moved to the destination (marker). An example of this state is shown in Figure 43. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0207] As the person taking the photograph moves the smartphone, the length of the arrow becomes shorter, and when an arbitrary point in the original image information, for example an arbitrary point (reference point) near the end of the image information on the opposite side from the direction in which the smartphone is being moved, reaches the position of the original marker, the length of the arrow becomes 0. At this time, a movement display is displayed indicating that the smartphone should be stopped, and a photograph is automatically taken with the smartphone camera. An example of this state is shown in Figure 44.

[0208] When this message is displayed, the person in charge of taking pictures keeps the smartphone still (S1000), and the photography processing unit 20 takes a picture again with the smartphone camera in the same manner as described above (S1010). At this time, the image information taken by the photography processing unit 20 with the smartphone camera is shown in Fig. 45(a).

[0209] Then, the image information captured by the image processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 performs correction processing to correct the image to a position facing directly (S1020). Figure 45(b) shows the image information after correction processing.

[0210] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing processing unit 22 (S1030). Fig. 46 shows the state in which the object areas and types have been detected for the image information after the correction processing of Fig. 45(b). Fig. 46 shows the state in which the areas and types of each of the objects "Shelf", "Faces", and "Shelf" have been detected from the bottom.

[0211] The photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether "Space Over Rack", which is the photography end condition, has been detected (S1040). In this case, "Space Over Rack", which indicates that photography has been completed up to the top of the display shelf, has not been detected, so the photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction in which the smartphone is moved to take a photograph, preferably on the object at the extreme end or in its vicinity (S1050). All or part of the area of ​​the object "Shelf" at the top of Figure 46 is set as the marker.

[0212] Then, the amount of movement of the angle of view from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the edge on the opposite side to the direction in which the smartphone is moved, to the marker is calculated (S1060), and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the smartphone display (S1070). This state is shown schematically in Figure 47.

[0213] Then, when the photographer moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the distance moved to the destination (marker). An example of this state is shown in Figure 48. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0214] When an arbitrary point (reference point) near the end of the image information on the opposite side to the direction in which the smartphone is being moved reaches the position of the original marker, the length of the arrow becomes 0. At that time, a movement display is displayed indicating that the smartphone should be stopped, and a photo is automatically taken with the smartphone camera. An example of this state is shown in Figure 49.

[0215] When this message is displayed, the person in charge of taking pictures keeps the smartphone still (S1000), and the photography processing unit 20 takes a picture again with the smartphone camera in the same manner as described above (S1010). At this time, the image information taken by the photography processing unit 20 with the smartphone camera is shown in Fig. 50(a).

[0216] Then, the image information captured by the image processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 performs correction processing to correct the image to a position facing directly (S1020). Figure 50(b) shows the image information after correction processing.

[0217] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing processing unit 22 (S1030). Figure 51 shows the state in which the object areas and types have been detected for the image information after the correction processing of Figure 50(b). Figure 51 shows the state in which the areas and types of each of the objects "Shelf", "Faces", "Shelf", and "Faces" have been detected from the bottom.

[0218] The photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether "Space Over Rack," which is the photography end condition, has been detected (S1040). In this case, "Space Over Rack," which indicates that photography has been completed up to the top of the display shelf, has not been detected, so the photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction of moving the smartphone's photography direction, preferably on the object at the extreme end or in its vicinity (S1050). All or part of the area of ​​the object "Faces" at the top of Figure 51 is set as the marker.

[0219] Then, the amount of movement of the angle of view from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the edge on the opposite side to the direction in which the smartphone is moved, to the marker is calculated (S1060), and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the smartphone display (S1070). This state is shown schematically in Figure 52.

[0220] When the photographer moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the distance moved to the destination (marker). An example of this state is shown in Figure 53. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0221] When an arbitrary point (reference point) near the end of the image information on the opposite side to the direction in which the smartphone is being moved reaches the position of the original marker, the length of the arrow becomes 0. At that time, a movement display is displayed indicating that the smartphone should be stopped, and a photo is automatically taken with the smartphone camera. An example of this state is shown in Figure 54.

[0222] When this message is displayed, the person in charge of taking pictures stops the smartphone (S1000), and the photography processing unit 20 takes a picture again with the smartphone camera in the same manner as described above (S1010). At this time, the image information taken by the photography processing unit 20 with the smartphone camera is shown in Fig. 55(a).

[0223] Then, the image information captured by the image processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 performs correction processing to correct the image to a position facing directly (S1020). Figure 55(b) shows the image information after correction processing.

[0224] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing processing unit 22 (S1030). Figure 56 shows the state in which the object areas and types have been detected for the image information after the correction processing of Figure 55(b). Figure 56 shows the state in which the areas and types of each of the objects "Faces," "Shelf," "Faces," and "SpaceOverRack" have been detected from the bottom.

[0225] Then, the photography processing unit 20 determines a photography end condition, for example, whether a predetermined object has been detected, using the area and type of the object detected by the object detection processing unit 23 (S1040). In this case, the object "SpaceOverRack" indicating that photography has been completed up to the top of the display shelf has been detected, so the processing in the photography processing unit 20 ends.

[0226] After the photographing process in the photographing processing unit 20 is executed as described above, the upload processing unit 26 sends the photographed image information (image information before correction) and the parameters for performing correction on the image information output by the correction processing unit 22 to the management server 3 (S910). That is, the upload processing unit 26 sends the image information and the parameters for performing correction of each of Figures 40(a), 45(a), 50(a), and 55(a) to the management server 3.

[0227] The upload processing unit 26 may send the corrected image information, that is, the image information of FIGS. 40(b), 45(b), 50(b), and 55(b), to the management server 3.

[0228] In the upload process, the upload processing unit 26 may send a shelf structure table corresponding to each piece of image information.

[0229] Then, when the image information reception processing unit 30 of the management server 3 receives each piece of pre-correction image information and the parameters for performing the correction uploaded from the portable communication terminal 2, the correction reproduction processing unit 31 performs a correction reproduction process on the uploaded pre-correction image information using the parameters for performing the correction process (S920). Therefore, the correction reproduction process is executed to transform each piece of pre-correction image information in Figures 40(a), 45(a), 50(a), and 55(a) into corrected image information using the parameters for performing the correction.

[0230] It should be noted that if the image information reception processing unit 30 receives the corrected image information, the correction reproduction processing in the correction reproduction processing unit 31 is not necessary.

[0231] Then, the product identification processing unit 320 in the management server 3 performs product identification processing on each corrected image information, and identifies the products displayed in the areas where products are displayed in each image information (areas labeled "Faces" or "Hanging Faces" (areas where it is determined that products are present)) (S930, S940).

[0232] That is, a product identification process is performed on each of the corrected image information of Figures 40(a), 45(a), 50(a), and 55(a), and the products displayed in the areas where the products are displayed in each of the image information are identified (S930, S940).

[0233] Similarly, the price reading processing unit 321 in the management server 3 performs a price reading process on each corrected image information and reads the price of each price tag in the area where the price tag is located in each image information (the area labeled "Shelf" or "HangingShelf" (the price display area where it is determined that a price tag is located)) (S935, S945).

[0234] By performing the above-described processing, it is possible to take appropriate photographs when photographing in a store, and to identify products displayed on display shelves.

[0235] If the shelf structure table corresponding to each image information is also uploaded during the upload process, the structure of a single display shelf can be identified by combining the shelf structure tables based on the tracking object. Therefore, based on the results of the product identification process and price reading process for each image information, it is also possible to identify which product is displayed on which shelf and how much the price tag says on it.

[0236] By executing the above-described processing, it is possible to execute processing similar to that in the first embodiment.

[0237] When performing the object detection process, the image capture processing unit 20 may also perform an out-of-view determination process, similar to Example 1. When performing the out-of-view determination process on the corrected image information, for example, if it is determined that the left and right sides of a display shelf are out of view, a guide to step back before taking a photo may be displayed on the smartphone display.

[0238] In the first to sixth embodiments, the order of processing can be changed as desired. Processing can be added or deleted without changing the spirit of the present invention. Furthermore, since the information processing system 1 of the present invention can execute image correction processing with a small processing load, a portable communication terminal 2 such as a smartphone or tablet computer with low processing power is preferable as the information processing device, and although this case has been described, it goes without saying that the information processing device may also be executed by a computer or server with high processing power.

[0239] Any of the embodiments may be combined as appropriate. [Industrial Applicability]

[0240] By using the information processing system 1 of the present invention, it is possible to reduce the number of image information to be processed while reducing the burden of photography. In addition, since it is possible to determine image information that is out of date at the store where the photography is performed, it is possible to rephotograph the image on the spot, thereby preventing a decrease in work efficiency. [Explanation of symbols]

[0241] 1: Information processing system 2: Portable communication terminal 20: Shooting processing section 21: Image information storage unit 22: Correction processing unit 23: Object detection processing unit 24: Cut-off detection processing unit 25: Image selection processing unit 26: Upload processing section 30: Image information reception processing unit 31: Correction reproduction processing section 32: Recognition processing section 220: Sensor information input reception processing unit 221: Angle estimation processing unit 222: Image transformation processing unit 223: Output processing section 320: Product identification processing unit 321: Price reading processing unit 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74:Communication equipment 75: Imaging equipment 76: Sensor device

Claims

1. An information processing system for image information obtained by photographing a display shelf, an image capturing unit for capturing an image of the display shelf; a recognition processing unit that performs a product identification process and / or a price reading process on the image information, The imaging processing unit Until the specified shooting end conditions are met, Correction processing is performed on the captured image information, When it is detected that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to movement of the portable communication terminal, the portable communication terminal performs a photograph or accepts a photographing operation. An information processing system comprising:

2. The imaging processing unit Until the shooting end condition using the object in the image information is met, performing a correction process on the captured image information; Detecting an object in the corrected image information; a predetermined location is set on or near an object that is located in a moving direction of the portable communication terminal among objects shown in the image information; When it is detected that a reference point in the captured image information or the corrected image information has moved to the predetermined position due to movement of the portable communication terminal, the portable communication terminal performs an image capture or accepts an image capture operation.

2. The information processing system according to claim 1, wherein:

3. The imaging processing unit Calculating the amount of movement from the reference point to the predetermined location, and displaying the movement according to the amount of movement and / or the direction of movement; 3. The information processing system according to claim 2.

4. The imaging processing unit includes: When it is detected that the reference point has moved to the predetermined location, a predetermined notification is made.

4. The information processing system according to claim 3.

5. The correction processing unit a sensor information input reception processing unit that receives input of pitch angle and roll angle information of the image capturing device or the information processing device and distance information to an arbitrary point on the image capturing object; an image transformation processing unit that performs image transformation processing on the image information, The image transformation processing unit calculating a regression line of a point cloud including a plurality of the points using the distance information and the information on the pitch angle and roll angle, and estimating information on a yaw angle of the imaging device or the information processing device; Using information on the estimated yaw angle, the image information is transformed so that it faces the image capture device.

2. The information processing system according to claim 1, wherein:

6. The image transformation processing unit distance information to an arbitrary point on the object to be photographed is converted into three-dimensional information, and the point is rotated based on pitch angle and roll angle information. The regression line of the point cloud when a point cloud including the arbitrary point is projected onto a horizontal plane is calculated, thereby estimating the yaw angle information.

6. The information processing system according to claim 5.

7. The correction processing unit an output processing unit that outputs correction parameters for performing image transformation processing on the image information; 6. The information processing system according to claim 5, further comprising:

8. The recognition processing unit executing the commodity identification process on the commodity area detected by the object detection processing unit, and / or executing the price reading process on the price display area detected by the object detection processing unit; 2. The information processing system according to claim 1, wherein:

9. The information processing system includes: an upload processing unit that uploads a part or all of the image information before correction and correction parameters for the image information selected by the image selection processing unit; a correction reproduction processing unit that reproduces the correction using part or all of the uploaded uncorrected image information and the correction parameters, The recognition processing unit performing a commodity identification process and / or a price reading process on a part or all of the corrected image information reproduced by the correction reproduction processing unit; 2. The information processing system according to claim 1, wherein:

10. Computer, an imaging processing unit for imaging the display shelves; An information processing program that functions as a recognition processing unit that performs a commodity identification process and / or a price reading process on image information, The imaging processing unit includes: Until the specified shooting end conditions are met, Correction processing is performed on the captured image information, When it is detected that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to movement of the portable communication terminal, the portable communication terminal performs a photograph or accepts a photographing operation. An information processing program characterized by:

11. A portable communication terminal used in an information processing system that photographs a display shelf and recognizes products and / or prices displayed in the image information, an imaging processing unit for imaging the display shelves; An information processing program that functions as an upload processing unit that uploads part or all of the captured image information or corrected image information in order to execute a commodity identification process and / or a price reading process using the image information, The imaging processing unit includes: Until the specified shooting end conditions are met, Correction processing is performed on the captured image information, When it is detected that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to movement of the portable communication terminal, the portable communication terminal performs a photograph or accepts a photographing operation. An information processing program that functions as a

Images