Information processing system

Abstract

The purpose of the present invention is to provide an information processing system used for identification / price reading of a commodity displayed on a display shelf.　The information processing system images a display shelf and processes image information. The information processing system comprises: an imaging processing unit that detects a movement amount and images a display shelf; a correction processing unit that performs correction processing on image information imaged by the imaging processing unit; an object detection processing unit that detects an object which appears in the image information that has been subjected to the correction processing; a cutoff determination processing unit that performs a cutoff determination using the presence / absence of the object detected from the image information and / or the positional relationship of the detected object, and that issues a predetermined warning if it is determined that cutoff has occurred; an image selection processing unit that selects, from a plurality of pieces of image information imaged by the imaging processing unit or of image information after the correction processing by the correction processing unit, image information to be subjected to commodity identification processing and / or price reading processing; and a recognition processing unit that executes the commodity identification processing and / or the price reading processing on the selected image information.

Description

Information Processing Systems

[0001] The present disclosure relates to an information processing system used to identify and read the prices of products displayed on shelves. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from Patent Application No. 2023-110352, filed July 4, 2023, the entire contents of which are incorporated herein by reference.

[0003] In convenience stores, supermarkets, and other stores, it is common for products to be displayed on shelves. Therefore, by displaying multiple products on a shelf, even if one product is purchased, another person can purchase the same product. Also, products displayed in large quantities in prominent locations attract attention and sell more than other products, creating a competitive advantage / inferiority relationship. Therefore, managing where and how many products are displayed on the shelf, as well as the prices of those products, is important in product sales strategies.

[0004] Therefore, by photographing the display shelves with a photographing device such as a camera and automatically recognizing the objects in the image information, it is possible to ascertain the display positions and numbers of products, the displayed products, and the prices written on the price tags (price cards).Technologies for this purpose are disclosed in Patent Documents 1 and 2 below.

[0005] JP-A-5-334409 JP-A-5-342230

[0006] When photographing products displayed on display shelves with a camera, it is common to be restricted by various photographing conditions, such as the space available for the photographing location, the presence of obstacles, etc. For example, if the width of the aisle between the display shelves where the products are displayed is narrow, it is not possible to photograph the display shelf with a single image, and one display shelf must be photographed with multiple images.

[0007] When identifying products displayed on a display shelf, processing is performed on a shelf-by-shelf basis, so a single display shelf is constructed using multiple pieces of image information, and product identification processing and price reading processing are performed based on this.

[0008] When photographing display shelves in sections, combining multiple images into a single image results in image distortion (steps and blurring) at the combined points, which affects the accuracy of product identification and price identification. Therefore, while not combining images is better for the accuracy of product identification and price reading, the photographer is required to take as few images as possible, which increases the workload of the photographer. Therefore, it is preferable for the photographer to take photographs mechanically, but this may result in an increase in the number of images taken. This increases the amount of image information to be processed, which leads to an overall increase in processing load. Therefore, there is a need to reduce the number of images to be processed while reducing the workload of the photographer.

[0009] Furthermore, when photographing display shelves in sections, there may be gaps between the image information, resulting in areas that are not captured. Or, the left and right sides of the display shelves may be cut off. In such cases, it is necessary to take photographs again, but these situations become apparent during the actual image analysis stage. Therefore, photographs must be taken again after the image analysis stage. However, in order to take photographs, it is necessary to go back to the store and take photographs again, which significantly reduces work efficiency.

[0010] In view of the above-mentioned problems, an information processing system for identifying products displayed on a display shelf has been disclosed.

[0011] An information processing system according to one aspect of the present disclosure is an information processing system for image information photographing a display shelf, and includes: a photography processing unit that detects the amount of movement and photographs the display shelf; a correction processing unit that performs correction processing on the image information photographed by the photography processing unit; an object detection processing unit that detects objects that appear in the image information that has been corrected; a photography determination processing unit that determines whether an object is cut off using the presence or absence of an object and / or the positional relationship of the detected object from the image information, and issues a specified warning if it determines that an object is cut off; an image selection processing unit that selects image information to be subjected to product identification processing and / or price reading processing from multiple pieces of image information photographed by the photography processing unit or image information after correction processing by the correction processing unit; and a recognition processing unit that performs the product identification processing and / or the price reading processing on the selected image information.

[0012] By using the above-mentioned information processing system, the person in charge of photography can mechanically photograph the display shelves according to information from the information processing system of the present disclosure, and the image information to be processed is appropriately selected, thereby reducing the burden of photography and the number of image information to be processed.

[0013] Furthermore, as described above, if the captured image information contains cut-off image information, a warning can be issued at the site where the image is being captured, thereby preventing a decrease in work efficiency, such as having to go back to the store to capture the image again.

[0014] An information processing system according to one aspect of the present disclosure is an information processing system for image information photographing a display shelf, and includes an imaging processing unit that detects the amount of movement and photographs the display shelf, a correction processing unit that performs correction processing on the image information photographed by the imaging processing unit, an object detection processing unit that detects objects that appear in the image information that has been corrected, an image selection processing unit that selects image information to be subjected to product identification processing and / or price reading processing from multiple pieces of image information photographed by the imaging processing unit or image information after correction processing by the correction processing unit, and a recognition processing unit that performs the product identification processing and / or price reading processing on the selected image information.

[0015] By using the above-mentioned information processing system, the person in charge of photography can mechanically photograph the display shelves while appropriately selecting the image information to be processed, thereby reducing the burden of photography and the number of image information to be processed.

[0016] In one aspect of the information processing system described above, the image selection processing unit can be configured to select any object in the image information to be processed as a tracking object, and non-adjacent image information that includes the tracking object as image information to be skipped, as image information to be subjected to the product identification processing and / or the price reading processing.

[0017] In one aspect of the information processing system described above, the image selection processing unit can be configured to select an object near the edge of the image information to be processed as a tracking object, and image information that includes the tracking object and is two or more positions away from the image information to be processed as skipped image information, and select it as the image information to be subjected to the product identification processing and / or the price reading processing.

[0018] By determining image information that includes a tracking object in the image information to be processed in this way, it is possible to select image information to skip to. This makes it possible to exclude the area between the image information to be processed and the image information to skip to from the processing target, thereby reducing the number of image information to be processed.

[0019] In one aspect of the information processing system described above, the image selection processing unit can be configured to use the position information of the tracking object in the image information to be processed and the amount of movement to determine whether the tracking object in the image information to be processed is included in the shooting range of the image information preceding the image information to be processed.

[0020] There are various processes for determining whether a tracking object is included in image information, but by using the process disclosed herein, it is possible to make a determination without matching image information of the tracking object area, thereby reducing the processing load.

[0021] The information processing system of the above aspect can be configured to include an out-of-view determination processing unit that performs out-of-view determination using the presence or absence of an object detected from the image information and / or the positional relationship of the detected object.

[0022] With the above-described configuration, it is possible to determine whether the captured image information includes cut-off image information at the store where the image is being captured.

[0023] In the information processing system according to the above aspect, the cut-off image determination processing section may be configured to display a predetermined warning when it determines that cut-off image information exists.

[0024] With the above-described configuration, if the captured image information contains cut-off image information, a warning can be issued at the site where the image is being captured, thereby preventing a decrease in work efficiency, such as having to go back to the store to capture the image again.

[0025] An information processing system according to one aspect of the present disclosure is an information processing system for image information of a display shelf, and includes an image processing unit that photographs the display shelf, a correction processing unit that performs correction processing on the image information photographed by the image processing unit, an object detection processing unit that detects objects that appear in the image information that has been corrected, an image cut-out determination processing unit that determines whether an object is cut-out using the presence or absence of an object and / or the positional relationship of the detected object detected from the image information, and issues a specified warning if it determines that an object is cut-out, and a recognition processing unit that performs the product identification processing and / or the price reading processing on the image information.

[0026] With the above-described configuration, if the captured image information contains cut-off image information, a warning can be issued at the site where the image is being captured, thereby preventing a decrease in work efficiency, such as having to go back to the store to capture the image again.

[0027] In one aspect of the information processing system described above, the cut-off determination processing unit can be configured to determine cut-off image information in the vertical direction based on the presence or absence of space above the shelf or stock product area, and area below the shelf, among the multiple captured image information.

[0028] In one aspect of the information processing system described above, the cut-off determination processing unit can be configured to determine the cut-off image information in the left-right direction by detecting whether, among the multiple captured image information, the area where the product is placed or the price display area is detected in the center in the left-right direction at a size equal to or larger than a predetermined size, and whether left-right cuts or left-right structural materials are within a predetermined range from the left and right edges of the image information.

[0029] The cut-off determination process can be performed in this manner.

[0030] In the information processing system according to the above aspect, the photographing processing unit may be configured to automatically photograph when it detects that the distance in the angle of view has moved by a predetermined value or more.

[0031] With the above-described configuration, the person in charge of photography can automatically take photographs by simply continuing to mechanically change the orientation of the photography device in a fixed direction, thereby reducing the burden on the person in charge of photography.

[0032] An information processing system according to one aspect of the present disclosure is an information processing system for image information of a display shelf, and includes an image processing unit that photographs the display shelf, and a recognition processing unit that executes the product identification process and / or the price reading process on the image information, wherein the image processing unit performs a correction process on the photographed image information until a predetermined photographing end condition is met, and when it detects that a reference point in the photographed image information or the corrected image information has moved to a predetermined location due to movement of a portable communication terminal, it performs a photograph or accepts a photographing operation.

[0033] With the above configuration, when taking a photo with a portable communication terminal, the photo is taken when the reference point moves to a predetermined position, or the photo is taken by accepting a photo operation, so the number of images taken can be reduced, which reduces the number of times correction processing, object detection processing, etc. are performed, and reduces the processing load.

[0034] In one aspect of the information processing system described above, the photography processing unit can be configured to perform correction processing on the photographed image information until a photography termination condition is met using the object depicted in the image information, detect the object depicted in the corrected image information, set a predetermined location at or near an object depicted in the image information that is in the direction of movement of the portable communication terminal, and perform photography or accept photography operations when it detects that a reference point in the photographed image information or the corrected image information has moved to the predetermined location due to the movement of the portable communication terminal.

[0035] As described above, a marker may be set on or near an object detected from captured image information, and processing may be performed.

[0036] In the information processing system of one aspect described above, the photography processing unit can be configured to calculate the amount of movement from the reference point to the specified location and display movement according to the amount of movement and / or the direction of movement.

[0037] By performing the above-described processing, the person in charge of photography can easily recognize how much and in what direction to move the portable communication terminal.

[0038] In the information processing system according to the above aspect, the photographing processing unit may be configured to issue a predetermined notification when it detects that the reference point has moved to the predetermined location.

[0039] As described above, for example, when it is detected that the portable communication terminal has moved to a specified location, a message is displayed indicating that the portable communication terminal should not be moved, and the person in charge of photography can operate the portable communication terminal in accordance with the instructions.

[0040] In the information processing system of one aspect described above, the correction processing unit includes a sensor information input reception processing unit that receives input of pitch angle and roll angle information of the image capture device or information processing device and distance information to an arbitrary point on an object to be captured, and an image transformation processing unit that performs image transformation processing on the image information, and the image transformation processing unit can be configured to calculate a regression line of a point cloud that includes a plurality of the points using the distance information and the pitch angle and roll angle information to estimate yaw angle information of the image capture device or information processing device, and to transform the image information so that it is positioned directly facing the image capture device using the estimated yaw angle information.

[0041] The above-described configuration enables correction processing without a large processing load, making correction processing possible even on a portable communication terminal with low performance, such as a smartphone.

[0042] In the information processing system of one aspect described above, the image transformation processing unit can be configured to convert distance information to an arbitrary point on the object to be photographed into three-dimensional information, rotate the point based on pitch angle and roll angle information, and calculate a regression line of a point cloud when the point cloud including the arbitrary point is projected onto a horizontal plane, thereby estimating the yaw angle information.

[0043] The correction process is preferably performed as described above.

[0044] In the information processing system of the above aspect, the correction processing unit can be configured to include an output processing unit that outputs correction parameters for performing image transformation processing on the image information.

[0045] Because the product identification process and price reading process require a high processing load, it is preferable to perform the process on a server with high processing power rather than on a portable communication terminal with low processing power, such as a smartphone. In this case, the image information to be subjected to the product identification process and price reading process is uploaded from the portable communication terminal to the server. However, when uploading the corrected image information, blank areas are included, increasing the amount of data. Therefore, by outputting correction parameters as disclosed herein, it is possible to upload the image information before correction and the correction parameters, thereby reducing the amount of data to be uploaded.

[0046] In one aspect of the information processing system described above, the recognition processing unit can be configured to perform the product identification process on the product area detected by the object detection processing unit, and / or to perform the price reading process on the price display area detected by the object detection processing unit.

[0047] As described above, by performing product identification processing / price display area processing on the product area / price display area detected by the object detection processing rather than performing product identification processing / price display area processing on the entire image information, the processing load can be reduced and processing can be speeded up.

[0048] In one aspect of the information processing system described above, the system may include an upload processing unit that uploads part or all of the pre-correction image information and correction parameters for the image information selected by the image selection processing unit, and a correction reproduction processing unit that reproduces the correction using part or all of the uploaded pre-correction image information and the correction parameters, and the recognition processing unit may be configured to perform product identification processing and / or price reading processing on part or all of the corrected image information reproduced by the correction reproduction processing unit.

[0049] Because the product identification process and price reading process require a high processing load, it is preferable to perform the process on a server with high processing power rather than on a portable communication terminal with low processing power, such as a smartphone. In this case, the image information to be subjected to the product identification process and price reading process is uploaded from the portable communication terminal to the server. However, when uploading the corrected image information, blank areas are included, increasing the amount of data. Therefore, by outputting correction parameters as disclosed herein, it is possible to upload the image information before correction and the correction parameters, thereby reducing the amount of data to be uploaded.

[0050] The information processing system disclosed above can be realized by loading and executing an information processing program of the present disclosure into a computer. That is, the information processing program causes a computer to function as an image capture processing unit that detects the amount of movement and captures an image of a display shelf, a correction processing unit that performs a correction process on image information captured by the image capture processing unit, an object detection processing unit that detects objects appearing in the corrected image information, an out-of-image determination processing unit that determines whether an object is detected in the image information and / or the positional relationship of the detected object and issues a predetermined warning if it determines that an object is out-of-image, an image selection processing unit that selects image information to be subjected to a commodity identification process and / or a price reading process from multiple pieces of image information captured by the image capture processing unit or image information corrected by the correction processing unit, and a recognition processing unit that performs the commodity identification process and / or the price reading process on the selected image information.

[0051] The information processing system disclosed above can be realized by loading and executing an information processing program of the present disclosure into a computer. That is, the information processing program causes a computer to function as an image capture processing unit that detects the amount of movement and captures an image of a display shelf, a correction processing unit that performs correction processing on image information captured by the image capture processing unit, an object detection processing unit that detects objects appearing in the corrected image information, an image selection processing unit that selects image information to be subjected to a commodity identification process and / or a price reading process from multiple pieces of image information captured by the image capture processing unit or image information corrected by the correction processing unit, and a recognition processing unit that executes the commodity identification process and / or the price reading process on the selected image information.

[0052] The information processing system disclosed above can be realized by loading and executing an information processing program of the present disclosure into a computer. That is, the information processing program causes a computer to function as an image capture processor that captures images of display shelves, a correction processor that performs correction processing on image information captured by the image capture processor, an object detection processor that detects objects appearing in the corrected image information, an out-of-view determination processor that determines whether an object is detected in the image information and / or the positional relationship of the detected object and issues a predetermined warning if it determines that an object is out-of-view, and a recognition processor that performs the product identification processing and / or the price reading processing on the image information.

[0053] The information processing system disclosed above can be realized by loading and executing the information processing program of the present disclosure into a computer. That is, the information processing program causes a computer to function as an image capture processing unit that captures images of display shelves and a recognition processing unit that performs the product identification processing and / or the price reading processing on the image information, wherein the image capture processing unit performs a correction processing on the captured image information until a predetermined image capture end condition is satisfied, and performs an image capture or accepts an image capture operation when it detects that a reference point in the captured image information or the corrected image information has moved to a predetermined position due to movement of a portable communication terminal.

[0054] The portable communication terminal used in the information processing system disclosed above can be realized by loading and executing the information processing program of the present disclosure into a portable communication terminal. That is, the information processing program causes a portable communication terminal used in an information processing system that photographs display shelves and recognizes the products and / or prices displayed in image information to function as an imaging processing unit that detects the amount of movement of an imaging device of the portable communication terminal to photograph the display shelves, a correction processing unit that performs correction processing on the image information photographed by the imaging processing unit, an object detection processing unit that detects objects appearing in the corrected image information, an image selection processing unit that selects image information to be subjected to a product identification process and / or a price reading process from multiple pieces of image information photographed by the imaging processing unit or from image information corrected by the correction processing unit, and an upload processing unit that uploads some or all of the selected image information to perform a product identification process and / or a price reading process using the image information selected by the image selection processing unit.

[0055] The portable communication terminal used in the information processing system disclosed above can be realized by loading and executing the information processing program of the present disclosure into a portable communication terminal. That is, the information processing program causes a portable communication terminal used in an information processing system that photographs display shelves and recognizes the products and / or prices displayed in the image information to function as an image capture processing unit that photographs the display shelves and an upload processing unit that uploads part or all of the photographed image information or corrected image information to perform a product identification process and / or a price reading process using the image information, in which the image capture processing unit performs a correction process on the photographed image information until a predetermined photographing end condition is satisfied, and performs a photograph or accepts a photographing operation when it detects that a reference point in the photographed image information or the corrected image information has moved to a predetermined position due to movement of the portable communication terminal.

[0056] By using the information processing system of the present disclosure, it is possible to reduce the number of image information to be processed while reducing the burden of photography. In addition, since it is possible to determine image information that is out of date at the store where the photography is being performed, it is possible to re-photograph the image on the spot, thereby preventing a decrease in work efficiency.

[0057] 1 is a block diagram schematically showing an example of the configuration of an information processing system of the present disclosure. FIG. 1 is a block diagram schematically showing an example of the hardware configuration of a computer used in the information processing system of the present disclosure. FIG. 2 is a flowchart showing an example of overall processing in the information processing system of the present disclosure. FIG. 3 is a flowchart showing an example of imaging processing in the information processing system of the present disclosure. FIG. 4 is a flowchart showing an example of image selection processing in the information processing system of the present disclosure. FIG. 5 is a flowchart showing an example of shelf structure analysis processing in the information processing system of the present disclosure. FIG. 6 is a flowchart showing an example of skip processing in the information processing system of the present disclosure. FIG. 7 is a diagram showing an example of object labels and their meanings. FIG. 8 is a diagram showing an example of out-of-view determination processing in the vertical direction, showing an example of a case where a SpaceOverRack area is detected as the space above the shelf. FIG. 9 is a diagram showing an example of out-of-view determination processing in the vertical direction, showing an example of a case where a Stock area is detected as a stock product area. FIG. 10 is a diagram showing an example of out-of-view determination processing in the vertical direction, showing an example of a case where a RackBottom area is detected as the space below the shelf. FIG. 11 is a diagram schematically showing a case where one display shelf is continuously photographed with a plurality of image information. FIG. 12 is a diagram schematically showing image selection processing. FIG. 13 is a diagram schematically showing another example of a case where one display shelf is continuously photographed with a plurality of image information. 10 is a diagram showing an example of image information before correction and image information after correction processing in the correction processing unit. FIG. 10 is a diagram showing an example of a shelf structure table for image information of #0. FIG. 10 is an example of a table showing rules for actions to be selected based on the relationship between objects. FIG. 10 is a diagram showing an example of a shelf structure table for image information of #5. FIG. 10 is a diagram showing an example of a shelf structure table for image information of #11. FIG. 10 is a diagram showing an example of a shelf structure table for image information of #14. FIG. 10 is a block diagram schematically showing an example of the configuration of the correction processing unit of Example 2. FIG. 10 is a flowchart showing an example of correction processing by the correction processing unit of Example 2. FIG. 10 is an example of image information to be input (image information before correction). FIG. 10 is a diagram schematically showing the relationship between a target surface and a three-dimensional space. FIG. 10 is an example of image information to be output (image information after correction). FIG. 10 is a front view of a state in which a point cloud after rotation is mapped onto a vertical plane. FIG. 10 is a top view of a state in which a point cloud after rotation is mapped onto a horizontal plane. FIG. 10 is a diagram schematically showing processing in an angle estimation processing unit.40( a ) is a diagram showing a schematic representation of processing in an image transformation processing unit; FIG. 40( b ) is a diagram showing a deviation in distance information when an object to be photographed has many recesses and projections; FIG. 40( c ) is a diagram showing a schematic representation of randomly selecting a point cloud from a matrix among point clouds on which distance information is plotted; FIG. 40( c ) is a diagram showing image information when recesses and projections are captured on the left and right sides of the original object to be photographed, and an abnormal value of distance information resulting therefrom; FIG. 40( c ) is a diagram showing a schematic representation of an example of a modified correction processing in Example 2; FIG. 40( a ) is a diagram showing an example of a flowchart of skip processing in Example 3; FIG. 40( c ) is a diagram showing an example of skip processing in Example 3; FIG. 40( c ) is a block diagram showing an example of a configuration of an information processing system in Example 6; FIG. 40( c ) is a conceptual diagram showing an example of processing in Example 6; 50(b) is a diagram showing an example of image information before and after correction when photographing is performed in the state of FIG. 44. FIG. 50(b) is a diagram showing an example of a state in which an object detection process has been performed on the image information of FIG. 45(b). FIG. 50(b) is a diagram showing an example of a state in which a movement display is superimposed. FIG. 50(b) is a diagram showing an example of a state in which the smartphone has been slightly moved in the movement direction. FIG. 50(b) is a diagram showing an example of a state in which a message is displayed to move the smartphone to a predetermined position and stop the movement. FIG. 50(b) is a diagram showing an example of an image information before and after correction when photographing is performed in the state of FIG. 54. ...FIG. 56 is a diagram showing an example of a state in which object detection processing has been performed on the image information of FIG. 55(b).

[0058] 1 shows a block diagram of an example of the overall processing functions of an information processing system 1 according to one embodiment of the present disclosure. The information processing system 1 preferably functions as a portable communication terminal 2, such as a smartphone, that has the functionality of an image capturing device 75.

[0059] The portable communication terminal 2 is a portable information processing device (computer) such as a smartphone or tablet computer that has the functionality of the image capturing device 75. It may also be a laptop computer. The external appearance and name of the portable communication terminal 2 are not important as long as it has the functionality necessary for the processing of the information processing system 1 of one embodiment of the present disclosure.

[0060] 2 schematically shows an example of the hardware configuration of the portable communication terminal 2 in the information processing system 1. The portable communication terminal 2 has a computing device 70 such as a CPU that executes the arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 that allows information to be input, a communication device 74 that transmits and receives the processing results of the computing device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN, an imaging device 75 such as a camera, and various sensor devices 76.

[0061] When the portable communication terminal 2 is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in portable communication terminals 2 such as tablet computers and smartphones, but are not limited to these.

[0062] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0063] The image capturing device 75 may be any device that captures image information using visible light or the like.

[0064] The sensor device 76 may include a sensor that detects the angle of view captured by the imaging device 75, a sensor that detects the pitch angle and roll angle of the portable communication terminal 2 or the imaging device 75, a distance measuring sensor that measures the distance from the imaging device 75 to the object being photographed, and the like, and it is desirable for some or all of these to be included, but this is not limited to these.

[0065] The functions of the means in the present disclosure are only logically distinct, and may be physically or practically in the same area. The order of the processes in the means in the present disclosure may be changed as appropriate. Part of the processes may also be omitted. For example, the process of determining the viewpoint direction, which will be described later, may be omitted. In this case, the process may be performed on image information that has not undergone the process of determining the viewpoint direction.

[0066] In the information processing system 1, functions are distributed between the management server 3 and the portable communication terminal 2. The way in which these functions are distributed can be changed as appropriate.

[0067] The portable communication terminal 2 in the information processing system 1 has an image capture processing unit 20, an image information storage unit 21, a correction processing unit 22, an object detection processing unit 23, an out-of-view determination processing unit 24, an image selection processing unit 25, and an upload processing unit 26.

[0068] The photography processing unit 20 photographs a subject to be photographed, for example, a display shelf in a store, using the photography device 75. When photography begins, the photographer moves the portable communication terminal 2 in a certain direction, and when the angle of view moves a certain distance, the sensor for detecting the angle of view in the sensor device 76 detects that the angle of view has moved a certain distance, and then the photography processing unit 20 photographs the subject. This process is repeated until photography is completed.

[0069] For example, the person in charge of photography fixes the portable communication terminal 2 in a fixed position while pointing the camera 75 of the portable communication terminal 2 toward the display shelf, and changes the orientation of the camera 75 of the portable communication terminal 2 so that the camera 75 takes a picture from above the display shelf to below. The photography processing unit 20 takes a picture at the start of photography, and then, when the sensor that detects the angle of view of the sensor device 76 detects that the angle of view has moved a fixed distance, for example, 1 / 10 of the vertical distance of the image information, the photography processing unit 20 takes a picture with the camera 75. This process is repeated until photography is completed.

[0070] This allows the photographing processing unit 20 to photograph a plurality of successive image information pieces from the top to the bottom of the display shelf, with the vertical positions of the angle of view shifted by 1 / 10 each, for example.

[0071] The image information captured by the image capture processing unit 20 is stored in the image information storage unit 21 .

[0072] The image information storage unit 21 stores image information captured by the image capture processing unit 20 .

[0073] The correction processing unit 22 performs correction processing on the image information captured by the image capturing processing unit 20 and stored in the image information storage unit 21 so that the image information is positioned in a normal position. The correction processing unit 22 may perform any processing as long as it can correct the image information so that the image information is positioned in a normal position. For example, it may perform a known keystone correction processing.

[0074] When photographing a display shelf in a store using the photographing processing unit 20, it may be difficult to ensure a sufficient distance between the display shelf to be photographed and the photographing device 75 due to circumstances such as other display shelves being located behind the photographer and narrow aisles. Furthermore, since the display shelf is located above or below the photographer's line of sight, photographing is often performed at an elevation or depression angle. As a result, photographing is not possible from a direct facing position. The correction processing unit 22 corrects the photographed image information so that it appears as if the photograph was taken from a direct facing position.

[0075] The correction processing unit 22 may generate image information by performing a correction process on the image information captured by the image capturing processing unit 20, or may output parameters (correction parameters) for transforming the image information so that it is positioned in a frontal view. For example, the correction processing unit 22 may output parameters (parameters (correction parameters) used for projective transformation) for generating image information corrected by projective transformation of the image information before correction so that it is positioned in a frontal view.

[0076] The object detection processing unit 23 detects the area and type (label) of an object (target) shown in the image information corrected by the correction processing unit 22. The process of detecting objects from image information can be performed by using deep learning on the image information to detect objects shown therein. In this case, the image information corrected by the correction processing unit 22 (corrected image information) is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the area and type of object shown in the image information are detected. The learning model can be a model in which various image information of a display shelf is provided with objects and their labels as ground truth data. An example of object labels and their meanings is shown in Figure 8.

[0077] The object detection processing in the object detection processing unit 23 uses deep learning, and can also compare sample data of various objects (image data of the object taken from one or more directions or data such as features based on the image data) with the image information corrected by the correction processing unit 22, and detect objects using the similarity.

[0078] The object detection process is not limited to these.

[0079] The cut-off determination processing unit 24 determines whether or not there is cut-off image information in the image information captured by the image capturing processing unit 20. The cut-off determination processing includes a process for determining cut-off in the vertical direction and a process for determining cut-off in the horizontal direction.

[0080] The process for determining whether the image information showing the top or bottom of the image information captured by the image capture processing unit 20 is cut off in the vertical direction is to determine whether a predetermined object has been detected in the object detection processing by the object detection processing unit 23, for example, whether the image information includes the space above the shelf (SpaceOverRack in the label in FIG. 8) or the stock product area (Stock in the label in FIG. 8), and the area below the shelf (RackBottom in the label in FIG. 8). If the predetermined object has not been detected, it is determined that there is image information that is cut off in the vertical direction.

[0081] An example of the process for determining whether or not the image is cut off in the vertical direction is shown in Figures 9 to 11. Figure 9 shows a case where a "SpaceOverRack" area is detected as the space above the shelf, Figure 10 shows a case where a "Stock" area is detected as the stock product area, and Figure 11 shows a case where a "RackBottom" area is detected as the space below the shelf.

[0082] The process for determining whether or not a portion of the image is cut off in the horizontal direction is based on the positional relationship of a predetermined object in the image information captured by the image capture processor 20, among the objects detected by the object detection processor 23 during object detection. For example, this can be determined by whether a stock product area (Stock in the label in FIG. 8 ), a shelf product area (Faces in the label in FIG. 8 ), a price display area for hanging displays (HangingShelf in the label in FIG. 8 ), a hanging product area (HangingFaces in the label in FIG. 8 ), or the like is detected at a certain size or larger in the horizontal center of the image information (however, these objects that exist within a certain range from the right or left edge are excluded from the determination as being on an adjacent shelf), or whether or not a left-right gap between the shelf levels (border in the label in FIG. 8 ), a vertical structural member of the shelf (pillar in the label in FIG. 8 ), or the like is within a certain range from the left or right edge. If a predetermined object is not detected at a certain size or larger in the horizontal center, or if the vertical structural member of the shelf is not within a certain range from the left or right edge, it is determined that the image information is cut off in the horizontal direction.

[0083] When the cut-off determination processing section 24 determines that there is cut-off image information, a predetermined warning such as that the image was not photographed properly or that the image was photographed in a cut-off state is displayed on the display device 72 of the portable communication terminal 2, and the user is prompted to photograph again.

[0084] The image selection processing unit 25 performs an image selection process that selects image information from the image information captured by the image capture processing unit 20 and uploads it to the management server 3 for product identification and / or price reading. That is, as in the present disclosure, when the image capture processing unit 20 continuously captures image information at a fixed amount of movement, such as a movement in the angle of view, it is possible that 10 or more images are captured when capturing an image of a single display shelf. This is schematically shown in FIG. 12. FIG. 12 shows a case in which 14 images of a single display shelf are continuously captured, with the capture range for each image indicated in parentheses. Note that the amount of movement is preferably the amount of movement in the angle of view, but is not limited thereto. For example, it may also be the amount of movement of the portable communication terminal 2.

[0085] 12, performing product and / or price tag identification processing on all image information photographed from one display shelf requires a large processing time and processing load, and is wasteful of processing. Therefore, from the image information photographed by the photographing processing unit 20 and stored in the image information storage unit 21, image information to be uploaded to the management server 3 and subjected to product and / or price tag identification processing is selected.

[0086] The image selection process in the image selection processing unit 25 analyzes the shelf structure of the first image data to be processed, from either the top or bottom, among the image data taken of the display shelf to be processed. The object detected at a predetermined position in the image data, preferably at the bottom or top, is designated as the tracking object. Then, the process skips to image data that includes the tracking object within the angle of view and is two or more images away from the image data (not adjacent to the image data), preferably the furthest away. The skipped image data is then selected as the image data to be processed, and the same process as described above is performed. This process is repeated until the final image data is reached. Figure 13 shows a schematic diagram of the image selection process.

[0087] For example, as shown in Fig. 14, suppose that the photography processing unit 20 takes 18 consecutive images (image information (1) to (18)) of a display shelf from bottom to top at each predetermined movement amount of the angle of view. The correction processing unit 22 performs correction processing on each of these image information, the object detection processing unit 23 performs object detection processing, and the cut-out determination processing unit 24 performs cut-out determination processing. Then, the image selection processing unit 25 uses the tracking object to select each of the image information (1), (5), (11), and (18) as the image information to be processed.

[0088] The upload processing unit 26 sends the image information (image information before correction) selected by the image selection processing unit 25 and the correction parameters for performing the correction on that image information output by the correction processing unit 22 in the correction processing to the management server 3. Furthermore, when sending the corrected image information to the management server 3, it is not necessary to send the correction parameters for performing the correction processing. For example, in the case of Figure 14, the image information before correction and the correction parameters for each of the image information (1), (5), (11), and (18) are sent to the management server 3.

[0089] The management server 3 includes an image information reception processing unit 30 , a correction reproduction processing unit 31 , and a recognition processing unit 32 .

[0090] The image information reception processing unit 30 receives image information uploaded from the upload processing unit 26 of the portable communication terminal 2 and correction parameters for performing correction on the image information output by the correction processing unit 22. For example, in the case of Fig. 14, the image information reception processing unit 30 receives the pre-correction image information and correction parameters for each of the image information (1), (5), (11), and (18). When the upload processing unit 26 uploads corrected image information, the correction parameters for performing the correction output by the correction processing are not required.

[0091] The correction reproduction processing unit 31 executes a correction process on the image information before correction using the parameters when the image information reception processing unit 30 receives the image information before correction and the parameters for performing the correction. When the image information corrected by the image information reception processing unit 30 is received from a portable terminal, the correction reproduction processing unit 31 is not necessary.

[0092] The recognition processing unit 32 performs a commodity identification process and / or a price reading process on the corrected image information. The recognition processing unit 32 includes a commodity identification processing unit 320 and a price reading processing unit 321.

[0093] The product identification processing unit 320 executes a process for identifying a product in the corrected image information. For example, the product may be identified using deep learning or image matching processing. Various known techniques can be used as the product identification processing unit 320.

[0094] When identifying products using deep learning, it is possible to first identify the area where the products are located (face area), and then perform a process to identify which products are displayed in that face area. The face area is an area that is the outline of the product or a predetermined area that includes the product, preferably a rectangular area.

[0095] In this case, the corrected image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the area (face area) where the product is located in the image information is identified. The learning model can be a model in which various image information of display shelves is provided with correct data for the area (face area) where the product is located. Furthermore, when performing product identification processing using deep learning, the object detection processing unit 23 may input areas in the corrected image information labeled "Faces" (shelf-mounted product area) or "HangingFaces" (hanging product area) (areas where the product is determined to be located) as processing targets, and perform face area identification processing using deep learning on those areas. The shelf-mounted product area and hanging product area are simply referred to as product areas.

[0096] Next, the image information of the face area is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and the product identification information shown in the image information of the face area is identified.The learning model can be one in which image information of various face areas is given correct answer data with product identification information (product identification information) as a label.The product identification information can be any information that can identify the product, such as the product name or JAN code.

[0097] Alternatively, the identification of the face area and the identification of the product identification information may be performed as a single process. In this case, for example, the following process can be executed. In this case, the corrected image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the face area captured in the image information and the identification information of the product in the face area are identified. The learning model can be various image information of a display shelf, to which corrective data is added, with the face area and product identification information as labels. The product identification information can be any information that can identify the product, such as the product name or JAN code. Furthermore, when performing product identification processing using deep learning, the object detection processing unit 23 can input areas (product areas) labeled with "Faces" or "Hanging Faces" in the corrected image information as processing targets, and perform processing to identify the face area and product identification information for those areas using deep learning.

[0098] Furthermore, when image matching processing is used, sample data (image information showing the appearance of the product from one or more directions or data such as feature quantities based thereon) depicting the appearance of various products may be compared with the corrected image information, and the product may be identified using the degree of similarity. When image matching processing is performed, the object detection processing unit 23 may perform image matching processing on areas (product areas) labeled with "Faces" or "Hanging Faces" in the corrected image information.

[0099] The price reading processor 321 executes a process of reading the price from the price tag in the corrected image information. For example, the price may be read using deep learning, character recognition processing, or image matching processing. Various known technologies can be used for the price reading processor 321.

[0100] When reading prices using deep learning, for example, the following process can be performed. In this case, corrected image information is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the price tag area captured in the image information is identified. The learning model can be a model in which various images of display shelves are provided with corrected data for price tag areas. Furthermore, when performing price tag identification processing using deep learning, the object detection processing unit 23 can input areas in the corrected image information labeled "Shelf" (shelf area with price display) or "HangingShelf" (hanging shelf price display area) as processing targets (hereinafter, areas with price display, such as shelf area with price display and hanging shelf price display area, are simply referred to as price display areas), and perform price tag identification processing using deep learning on those areas.

[0101] Furthermore, when image matching processing is used, sample data showing the appearance of various price tags (image information showing the appearance of a price tag from one or more directions or data such as features based thereon) may be compared with the corrected image information, and the price tag area may be identified using the degree of similarity. When image matching processing is performed, the object detection processing unit 23 may perform image matching processing on areas in the corrected image information labeled "Shelf" or "Hanging Shelf" (price display areas determined to contain price tags).

[0102] Character recognition processing is performed on the area of ​​the price tag identified using deep learning, image matching processing, etc., and the price is read.

[0103] The price reading process of the price reading processing unit 321 may involve identifying the area of ​​a price tag and then performing character recognition processing on that area, or it may involve inputting corrected image information and reading the price of the price tag shown in the image information. A learning model may be one in which correct data is provided for the price in the area of ​​a price tag in various image information photographs of a display shelf. When performing price reading processing using deep learning, the object detection processing unit 23 may input an area labeled "Shelf" or "Hanging Shelf" in the corrected image information (a price display area determined to contain a price tag) as the processing target, and perform price reading processing using deep learning on that area.

[0104] Next, an example of processing using the information processing system 1 of the present disclosure will be described with reference to the flowcharts of Figures 3 to 7. In this example, a case will be described in which the information processing system 1 of the present disclosure is used to identify products and read their prices based on image information of store shelves photographed with a camera (photography device 75) built into a smartphone (portable communication terminal 2).

[0105] First, a person in charge of photographing the store's display shelves performs a predetermined operation on the store's display shelves (the object to be photographed) using the camera on his or her smartphone, and the photography processing unit 20 then executes the photography process (S100).

[0106] The photographer performs a predetermined operation to activate a predetermined application software stored on the smartphone. This activation causes the photography processing unit 20 to activate the smartphone's camera. The photographer then points the photography device 75 of the portable communication terminal 2 toward the display shelves, holds the portable communication terminal 2 in a fixed position so as to minimize any change in the photography position, and begins photographing the display shelves from above to below, or from below to above (S200). That is, the photography processing unit 20 performs the first photograph when it detects that a photography start operation has been performed so as to capture the top of the display shelves (the space above the display shelves or the space where stock items are placed) or the bottom (the floor on which the display shelves are placed). The photographer then moves the portable communication terminal 2 in a fixed direction (upward or downward) to take a photograph (S210). As the photographing orientation of the portable communication terminal 2 is changed, the photographing processing unit 20 automatically photographs the display shelves (S230) when the sensor for detecting the angle of view in the sensor device 76 detects that the angle of view has moved a certain distance, for example, 1 / 10 of a second (S220). The photographing processing unit 20 repeats this process until a predetermined operation, for example, an operation to end photographing, is received (S240). This allows continuous image information of the display shelves to be photographed upward or downward from the fixed position of the portable communication terminal 2.

[0107] The person in charge of photography should move the smartphone from top to bottom or bottom to top while taking the photograph so that the photographing position does not change as much as possible. In addition, for consecutively photographed image information, it is preferable that identification information be assigned sequentially from the first photographed image information to the last photographed image information so that the order in which each image was photographed can be identified.

[0108] The photography processing unit 20 may also perform the following display processing to display instruction information, such as the orientation of the photograph, to the photographer. For example, the photography processing unit 20 of the portable communication terminal 2 displays the movement distance in the next angle of view as the length of an arrow on the display device 72 of the portable communication terminal 2. When the photographer changes the photography orientation of the portable communication terminal 2 in the direction of the arrow, the photography processing unit 20 calculates the movement distance in the angle of view due to the change in photography orientation and changes the displayed length of the arrow so that the arrow becomes shorter. When the length of the arrow reaches zero, the photography processing unit 20 stops displaying the arrow or displays a message such as "photography completed" on the display device 72, prompting the photographer to stop changing the photography orientation of the portable communication terminal 2. The photography processing unit 20 ends the photography process when it detects that the portable communication terminal 2 has stopped. In this case, the entire display shelf, from top to bottom or bottom to top, can be photographed in one or more images.

[0109] By performing the above-described photographing process by the photographing processing unit 20, it is possible to photograph a plurality of consecutive image information frames, each frame shifting by a certain distance from the top to the bottom of the display shelf or from the bottom to the top. The image information photographed by the photographing processing unit 20 is then stored in the image information storage unit 21. This is shown schematically in FIG. 12 or FIG. 14. FIG. 12 shows a case where 14 consecutive image information frames are photographed while changing the photographing direction of a single display shelf from top to bottom. FIG. 14 shows a case where 18 consecutive image information frames are photographed while changing the photographing direction of a single display shelf from bottom to top.

[0110] The correction processing unit 22 then executes a correction process on the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21, correcting the image information so that it is positioned in a normal position (S110). The correction process by the correction processing unit 22 may simply generate image information by correcting the image information itself, or may also output parameters for correcting the image information so that it is positioned in a normal position. If the parameters are also output, the amount of data uploaded during the upload process described below can be reduced by uploading the image information before correction and the parameters, compared to uploading corrected image information. For example, FIG. 15 shows an example of image information obtained by correcting the image information (1) in FIG. 14. FIG. 15(a) shows the image information before correction (1) in FIG. 14, and FIG. 15(b) shows the image information obtained by correcting the image information (1) in FIG. 14 by the correction processing unit 22. The correction process by the correction processing unit 22 is preferably performed on the image information captured by the image capture processing unit 20, but may be performed on only part of the image information rather than all of it.

[0111] After the correction processing by the correction processing unit 22 is completed, the object detection processing unit 23 detects the area and type of the object appearing in the image information corrected by the correction processing unit 22 (S120). That is, the image information corrected by the correction processing unit 22 is input to a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and objects appearing in the image information are detected.

[0112] The cut-off determination processing unit 24 also performs cut-off determination processing on the corrected image information, and determines whether or not there is cut-off image information (whether or not there is an error) in the image information captured by the image capturing processing unit 20 (S130).

[0113] The cut-off determining section 24 executes the process of determining cut-off in the vertical direction and the process of determining cut-off in the horizontal direction as described above.

[0114] As shown in Figures 9 to 11, the cut-off determination processing unit 24 determines whether a specific object has been detected in the object detection processing unit 23 in the image information captured by the image capture processing unit 20, and determines whether the image information includes, for example, the space above the shelf (SpaceOverRack in the label in Figure 8) or the stock product area (Stock in the label in Figure 8), and the area below the shelf (RackBottom in the label in Figure 8).

[0115] The out-of-image determination processing unit 24 determines whether or not an image is cut off in the horizontal direction based on the positional relationship in the image information of a predetermined object detected in the object detection processing unit 23 by the object detection processing unit 23 in the image information captured by the image capture processing unit 20. For example, the out-of-image determination processing unit 24 can determine whether a stock product area (Stock in the label in FIG. 8 ), a shelf product area (Faces in the label in FIG. 8 ), a price display area for hanging products (HangingShelf in the label in FIG. 8 ), a hanging product area (HangingFaces in the label in FIG. 8 ), etc. are detected at a certain size or larger in the horizontal center of the image information (however, these objects that exist within a certain range from the right end or a certain range from the left end are excluded from the determination as being on an adjacent shelf), or whether a left-right gap between the shelf levels (border in the label in FIG. 8 ), a vertical structural member of the shelf (pillar in the label in FIG. 8 ), etc. are within a certain range from the left or right end.

[0116] As described above, when the out-of-image determination processing unit 24 determines that there is cut-off image information (there is an error) (S130), the out-of-image determination processing unit 24 displays a predetermined warning on the display device 72 of the portable communication terminal 2, such as that the image was not captured properly or that the image was cut off, and urges the user to capture the image again. On the other hand, when the out-of-image determination processing unit 24 determines that there is no cut-off image information (there is no error), the image selection processing unit 25 executes image selection processing (S140).

[0117] The image selection processing unit 25 performs image selection processing to select image information from the image information captured by the image capture processing unit 20, which is uploaded to the management server 3 and subjected to product identification processing and / or price reading processing.

[0118] The image selection processing unit 25 initializes a shelf structure table that indicates the layout of the display shelf, and sets a parameter p to an initial value of p=0 (S400). Then, the image information having identification information indicating that the image was taken first is selected as the processing target (S410). For example, if the image processing unit 20 has taken images from above to below, the image information having the image of the top, i.e., the image information having identification information of "#0" in Fig. 13, is selected as the processing target.

[0119] The image selection processing unit 25 executes a shelf structure analysis process for the image information selected as the processing target (S420). The shelf structure analysis process is a process for detecting and associating areas where products are placed on a display shelf with areas where price tags are displayed, and executes the process shown in the flowchart of FIG. 6, for example.

[0120] In the shelf structure analysis process, first, the image information of parameter p is added as selected image information (S500). That is, the image information to be processed, in this case, image information "#0", is added as image information selected by the image selection processing unit 25.

[0121] Then, for the image information to be processed, i.e., image information "#0", the objects detected by the object detection processing unit 23 are extracted (S510). This is shown schematically in Fig. 13(a). In Fig. 13(a), the objects "Stock", "ShelfWoPC", "Faces", and "Shelf" are detected, and therefore these are extracted.

[0122] At this time, objects on the adjacent shelf are excluded (S520). In Fig. 13(a), there are no objects on the adjacent shelf, so the process is carried out as is.

[0123] The objects are then sorted in order from the direction in which they were photographed (S530). In FIG. 13, the photographs were taken from top to bottom, so the objects are sorted from top to bottom. In the case of FIG. 13(a), the objects are sorted in the order of "Stock," "ShelfWoPC," "Faces," and "Shelf." Each sorted object is then written to the shelf structure table (S540). FIG. 16 shows an example of a shelf structure table for the image information to be processed (image information #0).

[0124] Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf areas (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560). For example, the image selection processing unit 25 selects an action for the object based on the rule, an example of which is shown in Fig. 17, and executes the process of creating pairs.

[0125] Fig. 17(a) is a table showing names of actions to be performed based on the relationship between the current object and the object immediately below, and Fig. 17(b) is a table associating action names with action contents. The rules in Fig. 17 are merely an example and are not limited to these. It is preferable that the rules associate the relationship between adjacent objects and the action names with action contents, but any rules may be used.

[0126] In the case of the shelf structure table of FIG. 16 , the topmost object is "Stock" and the object immediately below it is "ShelfWoPC," so the table of FIG. 17 is referenced and an action with the action name "pairing" is performed, i.e., these two objects are paired. The next object is "ShelfWoPC" and the object immediately below it is "Faces," so the table of FIG. 17 is referenced and an action with the action name "OK" is performed, i.e., nothing is done. The next object is "Faces" and the object immediately below it is "Shelf," so the table of FIG. 17 is referenced and an action with the action name "pairing" is performed, i.e., these two objects are paired. The next object is "Faces," but there is no object immediately below it, so it is assumed that action selection processing for all objects in the image information to be processed has ended (S550), and the shelf structure analysis processing is terminated.

[0127] When the shelf structure analysis process is completed, if the image information for which the shelf structure analysis process has been performed is not the last image information (S430), the image selection processing unit 25 performs skip processing to select a skip destination image (S440).

[0128] The skip process is a process of finding other image information that includes a tracking object included in the image information to be processed, i.e., a process of finding image information to be skipped to. For example, the object at the bottom of the image information to be processed is set as the tracking object, and other image information that includes the area of ​​the tracking object at its top is found, and the found image information is set as the skip destination image information to be processed next.

[0129] In the skip process, a tracking object is determined in the image information (image information #0) to be processed by the parameter p, and its area is recorded (S600). For example, the bottommost object in image information #0, the "Shelf" object, is set as the tracking object, and its area is recorded.

[0130] The parameter p is incremented to p=1 (S610). Since the image information is not yet the last (S620), the image information of the identification information next to the image information of the identification information #0, i.e., the image information of the identification information #1, is selected, and the area of ​​the image information recorded in S600 is determined (S630). At this time, an image matching process may be performed between the image information of the tracking object area recorded in S600 and the image information of #1 to determine whether there is an area similar to the tracking object area. Furthermore, among the objects in the image information of #1, the area of ​​the object of the same type as the tracking object, "Shelf," may be matched. If the similarity is equal to or greater than a predetermined threshold, it is determined that the tracking object is present in the image information of #1.

[0131] If the image information with the incremented p contains an area corresponding to the tracking object (S640), p is incremented to p=2 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650). "There is an area corresponding to the tracking object" means that there is an area in the image information whose similarity to the tracking object area is equal to or greater than a predetermined threshold, and "there is no area corresponding to the tracking object" means that there is no area in the image information whose similarity to the tracking object area is equal to or greater than a predetermined threshold, for example, when there is no area corresponding to the tracking object, or when the area corresponding to the tracking object is only partial (cut off).

[0132] By repeating the above process, the image information to be skipped is searched for.

[0133] For example, when p = 6 (image information #6), if there are no more areas corresponding to the tracking object (if there are no more areas with a similarity greater than or equal to a predetermined threshold), p is decremented, p is set to 5, and the skip process is terminated (S650).

[0134] When the skip process is completed (S440), the image information of p=5, that is, the image information of #5, is selected as the processing object (S410), as shown in FIG.

[0135] Then, a shelf structure analysis process is performed on the image information for p=5, image information #5 (S420), and image information #5 is added as selected image information (S500). Then, the process from S510 onward is performed on image information #5 in the same manner as described above, extracting objects from image information #5 (S510), excluding objects on adjacent shelves (S520), and sorting them in order (S530). The objects are then written into the shelf structure table. An example of the shelf structure table for image information #5 is shown in FIG. 18. Then, the image selection processing unit 25 creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0136] After the shelf structure analysis process is performed on the image information of #5, the skip process is performed on the image information of #0 (S440).

[0137] The image selection processing unit 25 records the area of ​​the bottommost "Shelf" object as the tracking object in the image information of #5 (S600). Then, p is incremented to p = 7 (S610). Because p ≤ 14, this is not the last image information (S620). Therefore, image matching processing is performed between the image information of the tracking object area recorded in S600 (image information of the bottommost "Shelf" area in the image information of #5) and the image information of the "Shelf" object area in the image information of #7, and it is determined whether there is an area similar to the tracking object area (S630).

[0138] If the incremented image information p=7 (image information #7) contains an area corresponding to the tracking object (S640), p is incremented to p=8 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650).

[0139] By repeating the above process, the image information to be skipped is searched for.

[0140] For example, when p = 12 (image information #12), if there are no more areas corresponding to the tracking object (if there are no more areas with a similarity greater than or equal to a predetermined threshold), p is decremented, p is set to 11, and the skip process is terminated (S650).

[0141] When the skip process is completed (S440), the image information of p=11, that is, the image information of #11, is selected as the processing object (S410), as shown in FIG.

[0142] Then, the image information of p=11 and the image information of #11 are subjected to shelf structure analysis processing (S420), and the image information of #11 is added as selected image information (S500). Then, the image information of #11 is subjected to processing from S510 onward, as described above, to extract objects from the image information of #11 (S510), exclude objects on adjacent shelves (S520), and sort them in order (S530). The objects are then written into the shelf structure table. An example of the shelf structure table for the image information of #11 is shown in FIG. 19. The image selection processing unit 25 then creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0143] After the shelf structure analysis process is performed on the image information of #11, the skip process is performed on the image information of #5 (S440).

[0144] The image selection processing unit 25 records the area of ​​the bottommost "Faces" object as the tracking object in the image information of #11 (S600). Then, p is incremented to p=12 (S610). Because p≦14, this is not the last image information (S620). Therefore, image matching processing is performed between the image information of the tracking object area recorded in S600 (image information of the bottommost "Faces" area in the image information of #11) and the image information of the "Faces" object area in the image information of #12, and it is determined whether there is an area similar to the tracking object area (S630).

[0145] If the incremented image information p=12 (image information #12) contains an area corresponding to the tracking object (S640), p is incremented to p=13 (S610). On the other hand, if there is no area corresponding to the tracking object (S640), p is decremented (S650).

[0146] By repeating the above process, the image information to be skipped is searched for.

[0147] For example, when p = 14, there is an area corresponding to the tracking object up to (image information #14), and if p is incremented to p = 15 (S610), p ≤ 14 is obtained, so p is deemed to have passed the last image information (S620), p is decremented to p = 14, and the skip process is terminated (S650).

[0148] When the skip process is completed (S440), the image information of p=14, that is, image information #14, is selected as the processing object (S410), as shown in FIG.

[0149] Then, the shelf structure analysis process is performed on the image information of p=14 and the image information of #14 (S420), and the image information of #14 is added as selected image information (S500). Then, the process from S510 onward is performed on the image information of #14, as described above, to extract objects from the image information of #14 (S510), exclude objects on adjacent shelves (S520), and sort them in order (S530). The objects are then written into the shelf structure table. An example of the shelf structure table for the image information of #14 is shown in FIG. 20. The image selection processing unit 25 then creates pairs of product display areas (Faces, Stock, HangingFaces) and shelf levels (Shelf, ShelfWoPC, HangingShelf) for the areas of adjacent objects (S550, S560).

[0150] When the shelf structure analysis process is performed on the image information of #14, since this is the last image information (S430), the image selection processing section 25 ends the image selection process.

[0151] By performing the above-described processing, the image selection processing unit 25 can select each of the image information #0, #5, #11, and #14 in FIG. 12 as image information to be uploaded.

[0152] After the image selection process is completed, the upload processing unit 26 sends the image information (image information before correction) selected by the image selection processing unit 25 and the parameters for performing correction on that image information output by the correction processing unit 22 in the correction process to the management server 3 (S150). That is, the upload processing unit 26 sends the image information before correction for each of #0, #5, #11, and #14 and the parameters for performing correction to the management server 3.

[0153] The upload processing unit 26 may send the corrected image information of #0, #5, #11, and #14 to the management server 3.

[0154] In the upload process, the upload processing unit 26 may send a shelf structure table corresponding to each piece of image information.

[0155] Then, when the image information reception processing unit 30 of the management server 3 receives the pre-correction image information and the parameters for performing the correction uploaded from the portable communication terminal 2, i.e., the pre-correction image information of #0, #5, #11, and #14 and the parameters for performing the correction, the correction reproduction processing unit 31 performs a correction reproduction process on the uploaded pre-correction image information using the parameters for performing the correction process (S160). Therefore, the pre-correction image information of #0, #5, #11, and #14 is transformed into corrected image information using the parameters for performing the correction. The correction reproduction process is similar to the correction process in the portable communication terminal 2, but since it simply transforms the pre-correction image information using the parameters (projective transformation), it can be performed by arithmetic processing, allowing for high-speed processing.

[0156] It should be noted that if the image information reception processing unit 30 receives the corrected image information, the correction reproduction processing in the correction reproduction processing unit 31 is not necessary.

[0157] Then, the product identification processing unit 320 in the management server 3 performs a product identification process on each corrected image information, and identifies the products displayed in the areas where products are displayed in each image information (areas labeled "Faces" or "Hanging Faces" (areas where it is determined that products are present)) (S170, S180).

[0158] That is, a product identification process is performed on each of the corrected image information #0, #5, #11, and #14, and the products displayed in the areas where the products are displayed in each image information are identified (S170, S180).

[0159] Similarly, the price reading processing unit 321 in the management server 3 executes a price reading process on each corrected image information, and reads the price of each price tag in the area where the price tag is located in each image information (the area labeled "Shelf" or "HangingShelf" (the price display area where it is determined that a price tag is located)) (S175, S185).

[0160] By performing the above-described processing, it is possible to take appropriate photographs when photographing in a store, and to identify products displayed on display shelves.

[0161] If the shelf structure table corresponding to each image information is also uploaded during the upload process, the structure of a single display shelf can be identified by combining the shelf structure tables based on the tracking object. Therefore, based on the results of the product identification process and price reading process for each image information, it is also possible to identify which product is displayed on which shelf and how much the price tag says on it.

[0162] In the above-described first embodiment, various known correction processing techniques capable of executing correction processing to achieve a head-on position have been described as being usable as the correction processing in the correction processing unit 22. However, a case where correction processing with a reduced processing load can be realized can also be described. This allows correction processing to be executed with a reduced load even on a portable communication terminal 2 with low processing power, such as a smartphone.

[0163] An example of the configuration of the correction processing unit 22 in this embodiment is shown in FIG. 21, and an example of the correction processing in the correction processing unit 22 is shown in the flowchart of FIG.

[0164] In this embodiment, the correction processing unit 22 regards the front surface of the object to be photographed as a vertical rectangle (including a square) in three-dimensional space (this surface is called the object surface), and performs correction processing by estimating the positional relationship between the portable communication terminal 2 and the object surface in three-dimensional space using sensor information detected by the sensor device 76 of the portable communication terminal 2. Fig. 23 shows an example of image information (image information before correction) photographed by the photographing device 75.

[0165] The correction processing unit 22 includes a sensor information input reception processing unit 220 , an angle estimation processing unit 221 , an image transformation processing unit 222 , and an output processing unit 223 .

[0166] The sensor information input reception processing unit 220 receives input of information from the sensor device 76, which detects the pitch angle and roll angle of the portable communication terminal 2 or the image capturing device 75. It also receives input of distance information between the image capturing device 75 and the image capturing target from a sensor (ranging sensor) that measures the distance to the image capturing target. As the distance information, it obtains distance information from the image capturing device 75 to each point on the image capturing target.

[0167] The angle estimation processing unit 221 converts the distance information to each point on the object to be photographed, which is received by the sensor information input reception processing unit 220, into three-dimensional information, and rotates this point cloud based on the roll angle and pitch angle. Because the object surface is vertical, the result of the rotation is a plane that is roughly perpendicular to the xz plane, as shown in Figure 24. Then, by calculating a regression line of these points on the xz plane, the perpendicular line drawn from the image capture device 75 to this line is estimated as the yaw angle.

[0168] The image transformation processing unit 222 uses the yaw angle estimated by the angle estimation processing unit 221 to transform the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21 so that the image information is positioned directly facing the image capture device 75. The shape of the target rectangle of the target surface is determined by assuming four vectors with the image capture device 75 as endpoints, which represent the angle of view of the image capture device 75 in three-dimensional space. The vectors are then rotated based on the roll angle and pitch angle input received by the sensor information input reception processing unit 220, and the angle estimation processing unit 221 calculates four intersections with a vertical plane tilted to the left or right by the yaw angle estimated by the angle estimation processing unit 221. Then, the image information captured by the image capture processing unit 20 is subjected to a projective transformation process such as OpenCV using the calculated four intersections, thereby performing image transformation processing and correcting the image information from a directly facing position. FIG. 25 shows an example of image information corrected from the image information of FIG. 23 .

[0169] The output processing unit 223 outputs the image information corrected by the correction processing unit 22. For example, the output processing unit 223 outputs the corrected image information as shown in Fig. 25. The output processing unit 223 may also output parameters for correction by the correction processing unit 22, i.e., parameters for performing projective transformation processing.

[0170] Next, the correction processing of the correction processing unit 22 in this embodiment will be described with reference to the flowchart of FIG.

[0171] First, a photographer photographs the store's display shelves (the object to be photographed) using the camera on a smartphone that the photographer carries. The photography processing unit 20 stores the image information captured by the smartphone camera in the image information storage unit 21 (S700). An example of this image information is shown in FIG. 23. The image information in FIG. 23 was captured from a diagonally downward angle of the display shelves, looking upward (at an elevation angle), and the display shelves are slightly rotated to the left (the pitch angle is 18.8 degrees upward, and the roll angle is 4.6 degrees counterclockwise).

[0172] In addition, the sensor information input reception processing unit 220 receives input of information on the pitch angle and roll angle of the smartphone or camera when capturing image information using the imaging device 75, as measured by the sensor device 76, and information on the distance to each point on the object being photographed (S700).

[0173] Distance information may be acquired from any number of locations, for example, from several tens to a hundred locations, for the object being photographed. Note that the number of locations from which distance information is acquired may be any number as long as sufficient accuracy can be obtained.

[0174] The angle estimation processing unit 221 plots the distance information received as input by the sensor information input reception processing unit 220 in three-dimensional space and converts the distance information into a point cloud in the three-dimensional space (S710).The angle estimation processing unit 221 then rotates this point cloud using the pitch angle and roll angle information received as input by the sensor information input reception processing unit 220 (S720).

[0175] To generate corrected image information (trapezoid-corrected image information) in which the products displayed on the display shelves on the target surface are upright and the shelves are horizontal, i.e., to correct the target surface in the image information captured by the image capture processing unit 20 so that it is positioned directly facing the camera, it is necessary to calculate the degree to which the target surface is tilted from the orientation facing the camera. Therefore, the point cloud after rotation in S720 is mapped onto a horizontal plane (x-z plane), and a regression line is calculated using a robust estimation algorithm such as RANSAC (S730). Figure 26 is a front view (x-y plane in Figure 24) of the point cloud after rotation in S720 mapped onto a vertical plane, and Figure 27 is a top view (x-z plane in Figure 24) of the point cloud after rotation in S720 mapped onto a horizontal plane. Figure 28 also shows a schematic diagram of the processing in the angle estimation processing unit 221.

[0176] Then, the angle (yaw angle) of the target surface tilted from the optical axis of the camera can be calculated from this regression line. Specifically, if the slope of the regression line of the point cloud is a, then the slope a of the regression line is converted to a yaw angle by calculating Equation 1 (S740). That is, the arctangent is calculated using the slope a as an argument. (Equation 1) Yaw angle = a tan(a)

[0177] Since the angle estimation processing unit 221 can estimate the yaw angle through the above processing, the image transformation processing unit 222 uses the estimated yaw angle to transform the image information captured by the photography processing unit 20 and stored in the image information storage unit 21 so that the image information is positioned directly facing the photography device 75. This processing is schematically shown in Figure 29.

[0178] Since the angle estimation processing unit 221 has estimated the yaw angle and the positional relationship between the camera and the target surface in three-dimensional space is now known, the image transformation processing unit 222 sets four lines (vectors) in three-dimensional space connecting the four vertices formed by the horizontal plane, the camera position, and the camera's angle of view (S750).Then, these four lines (vectors) are rotated using the roll angle and pitch angle accepted as input by the sensor information input acceptance processing unit 220 and the yaw angle estimated by the angle estimation processing unit 221 (S760).

[0179] The image transformation processing unit 222 calculates a quadrangle of a cross section obtained by cutting the four straight lines rotated in S760 with the target plane (S770). The image transformation processing unit 222 performs projective transformation on the image information captured by the image capture processing unit 20 and stored in the image information storage unit 21 with respect to this target quadrangle to generate corrected image information (S780). The image information obtained by performing the correction processing on the image information in Figure 23 by the correction processing unit 22 is the image information in Figure 25.

[0180] By executing the above processing, the processing of the correction processing unit 22 can be executed.

[0181] The output processing unit 223 then outputs the image information (e.g., FIG. 25 ) corrected by the correction processing unit 22. The output processing unit 223 may also output parameters for projective transformation of the target rectangle as parameters to be used during correction. By outputting the parameters, the upload processing unit 26 can reproduce the correction in the correction reproduction processing unit 31 of the management server 3 by sending the image information before correction and the parameters instead of the image information after correction, thereby reducing the amount of data to be uploaded.

[0182] In addition to the above-described processing by the correction processing unit 22, the following processing can also be performed.

[0183] That is, when the object being photographed is uneven, such as a display shelf displaying products, the unevenness of the object being photographed has a much stronger correlation in the horizontal direction than in the vertical direction. Therefore, if the left-right direction of the camera 75 is slightly misaligned with the horizontal direction of the display shelf, the distance information for horizontally arranged objects is likely to exhibit a "step-like difference," where the objects appear close up until halfway and far away from there. If several such rows are mixed into the acquired distance information, the estimated results will be tilted relative to the actual object surface. This is schematically shown in Figure 30. Figure 30(a) shows image information captured of an uneven display shelf, such as a display shelf displaying products, as the object being photographed. Figure 30(a) shows that the position (point) from which distance information is acquired may be the position of a product located in the foreground or the position of a shelf with no products in the background. In this case, the point cloud positional relationship shown in Figure 30(b) is obtained, which is likely to result in tilted estimated results.

[0184] Therefore, in the correction process of the second embodiment, the target surface may be further divided into matrices of a predetermined size, and distance information of points scattered pseudo-randomly in each matrix may be acquired. This makes it possible to randomly select points from the matrix among the points plotted with distance information when converting the distance information into a point cloud in three-dimensional space in S710. This is schematically shown in FIG. 31.

[0185] The above processing can improve the accuracy of the estimation result.

[0186] Furthermore, the correction processing unit 22 may perform the following processing.

[0187] That is, when the object being photographed is a display shelf, the photographer typically stands near the center of the display shelf and takes a photograph so that the entire left and right sides of the display shelf are captured. In this case, there is a possibility that parts of adjacent display shelves will appear near the left and right edges of the captured image information. Figure 32 shows an example of image information illustrating this. In such image information, if the left and right display shelves protrude forward or are recessed backward compared to the display shelf that was originally being photographed, the regression line in S730 may be tilted. In the captured image information shown in Figure 32(a), the display shelf that appears to the right of the display shelf that was originally being photographed protrudes forward. Therefore, when the distance information is plotted in three-dimensional space, an abnormal value appears on the right side, as shown in the top view (x-z plane) of Figure 32(b). If these point clouds are processed as they are, it will affect the slope of the regression line.

[0188] Therefore, in the correction process in the correction processing unit 22 of the second embodiment, when a regression line is calculated using a process such as RANSAC, the following process may be executed.

[0189] First, using a process such as RANSAC, a predetermined number of points are randomly selected from the rotated point cloud in step S720 to determine a regression line. Then, for all point clouds, the number of points whose error from the regression line falls within a threshold is used as an evaluation function, and this operation is repeated multiple times. The regression line with the largest number of points is selected as the final regression line. For example, when determining a regression line from a point cloud of 100 points, a regression line is determined by selecting three points as the predetermined number of points, and sufficient accuracy can be achieved by repeating this process 500 times.

[0190] Furthermore, this evaluation function can be modified to lower the weight when selecting points on the outer edge, thereby reducing the influence of irregularities in the display shelves that are reflected near the left and right edges. For example, the weighting can be modified by counting points on the outer edge as 0.5 instead of 1.

[0191] More specifically, as shown in Figure 33, the point cloud acquisition section is divided into a predetermined number of sections, for example, four, in the x-axis direction, and the weighting of the number of points in the outer sections is changed depending on the position on the outer edge.

[0192] The correction processing in the correction processing unit 22 in the second embodiment can be applied not only to image information captured at an elevation angle or depression angle, but also to image information captured from the left and right directions.

[0193] In the skip processing of the image selection processing unit 25 in the first embodiment described above, the image information to be skipped is identified by using a process of comparing the image information in the tracking object area with similar image information.

[0194] Therefore, since comparing image information would increase the processing load, a case in which the image information comparison process is not used will be described in this embodiment 3. An example of the process in this case is shown in the flowchart of Fig. 34. Note that the process up to the skip process in the image selection processing unit 25 is the same as in embodiment 1 or 2.

[0195] A tracking object is determined in the image information (image information #0) to be processed with parameter p, and the position information (coordinate information) of the rectangle that constitutes that area is recorded (S800). For example, the bottommost object in image information #0, the "Shelf" object, is set as the tracking object, and the position information of the rectangle that constitutes that area is recorded. The position of the tracking object has been identified by the object detection process in object detection processing unit 23.

[0196] The parameter p is incremented to p=1 (S810). Since the image information is not yet the last (S820), the image information with the identification information next to the image information with the identification information #0, i.e., the image information with the identification information #1, is selected, and the amount of movement (movement distance) of the angle of view from the image information with the identification information #0 to the image information with the identification information #1 is determined (S830). Note that the amount of movement of the angle of view is the amount of movement used automatically by the imaging processing unit 20 for imaging. For example, the amount of movement is 1 / 10 of the angle of view.

[0197] Then, the coordinate value corresponding to the amount of movement in S830 is added to the position information of the rectangle that constitutes the area of ​​the tracking object recorded in S800 (S840). This makes it possible to identify where the tracking object in the image information of #0 is located in the image information of #1.

[0198] If the rectangular area that constitutes the tracking object is included in the image information in #1, it is determined that the tracking object is present in the image information in #1.

[0199] If the image information of the incremented p contains an area corresponding to the tracking object (S850), p is incremented to p=2 (S810). On the other hand, if there is no area corresponding to the tracking object (S850), p is decremented (S860).

[0200] By repeating the above process, the image information to be skipped is searched for.

[0201] FIG. 35 schematically illustrates the process of determining whether a tracking object is included in image information indicated by parameter p. FIG. 35(a) is a diagram schematically illustrating the amount of movement of the angle of view between image information for p=n and image information for p=n+1. One movement results in an amount of movement of the angle of view (α, β). Therefore, the position information (x, y) of a rectangle constituting the area of ​​the tracking object in the image information for p=n becomes the position information (x+α, y+β) of a rectangle constituting the area of ​​the corresponding tracking object in the image information for p=n+1. FIG. 35(b) illustrates this diagram. Therefore, it is sufficient that the position coordinates (x+α, y+β) of the vertices of the rectangle in the image information for p=n+1 are included in the area of ​​the image information for p=n+1. Although only one point is shown here, it is possible to determine whether the tracking object is included in the image information of p by determining that four points that make up a rectangle, or two points that make up the bottom edge (when moving from top to bottom), or two points that make up the top edge (when moving from bottom to top) are included.

[0202] By performing the above processing, it is possible to identify the image information to be skipped without performing a process of comparing the image information in the tracking object area with similar image information, thereby reducing the processing load on the image selection processing unit 25.

[0203] In addition, if the corrected image information is not used in the skip processing, the shape and position information of the tracking object and the position information of the image information of the parameter p may be calculated using the yaw angle identified by the correction processing unit 22 and the roll angle and pitch angle detected by the sensor device 76, and then processing may be performed.

[0204] As a variation of Examples 1 to 3, the upload processing unit 26 may be configured to cut out and upload image information of areas in the corrected image information where it is determined that there is a product (areas labeled "Faces" or "Hanging Faces" by the object detection processing unit 23) and image information of price display areas in the corrected image information where it is determined that there is a price tag (areas labeled "Shelf" or "Hanging Shelf" by the object detection processing unit 23), rather than uploading the corrected image information or the uncorrected image information and correction parameters.

[0205] In this case, the management server 3 does not need to be provided with the correction reproduction processing unit 31 .

[0206] The product identification processing unit 320 of the management server 3 performs a product identification process similar to that of Example 1 on the image information of an area in the corrected image information received from the portable communication terminal 2 by the image information reception processing unit 30 where it is determined that a product is present.

[0207] In addition, the price reading processing unit 321 of the management server 3 performs a price reading process similar to that of Example 1 on the image information of the price display area in which it is determined that a price tag is present in the corrected image information received from the portable communication terminal 2 by the image information reception processing unit 30.

[0208] This allows the amount of image information data to be uploaded to be further reduced.

[0209] In the photography processing unit 20 of the first to fourth embodiments described above, a single display shelf is continuously photographed using a plurality of pieces of image information, but if the aisle width is sufficiently wide so that a single display shelf can be photographed using a single piece of image information, the photography processing unit 20 may photograph a single display shelf using a single piece of image information. In this case, there is no need to perform image selection processing in the image selection processing unit 25. That is, the correction processing in the correction processing unit 22, such as in the first or second embodiment, is performed on the image information photographed by the photography processing unit 20, and the object detection processing in the object detection processing unit 23 is performed on the corrected image information.

[0210] After the cut-off determination processing is performed in the cut-off determination processing unit 24, the upload processing unit 26 uploads the corrected image information or the uncorrected image information and the correction parameters. Then, in the management server 3, the correction reproduction processing unit 31 performs the correction reproduction processing, the product identification processing unit 320 performs the product identification processing, and the price reading processing unit 321 performs the price reading processing.

[0211] Furthermore, when performing the same processing as in Example 4, after the cut-out determination processing is performed by the cut-out determination processing unit 24, the image information of the area in the corrected image information where it is determined that a product is present and the image information of the price display area where it is determined that a price tag is present are uploaded by the upload processing unit 26. Then, in the management server 3, the product identification processing unit 320 performs the product identification processing, and the price reading processing unit 321 performs the price reading processing.

[0212] A modified example of the photography processing unit 20 will be described. In the photography processing unit 20 of Example 1, when photography starts, the photographer moves the portable communication terminal 2 in a certain direction, and when the angle of view moves a certain distance, the sensor for detecting the angle of view in the sensor device 76 detects that the angle of view has moved a certain distance and photographs the image, and this process is repeated until the photographing is completed. In this case, if the photographer quickly moves the photographing direction of the portable communication terminal 2, blurring may occur when photographing with the photographing device 75, and the image may not be captured to an extent that allows for analysis of image information.

[0213] Therefore, in the photographing processing unit 20 of this embodiment, a case will be described in which, in addition to the photographing processing in the photographing processing unit 20, the photographing processing unit 20 also includes the out-of-view determination processing in the object detection processing unit 23 and the out-of-view determination processing unit 24. An example of a block diagram of the configuration of the information processing system 1 in this case is shown in Fig. 36.

[0214] In the photography processing unit 20 of this embodiment, after photography is performed by the photography device 75, the correction processing unit 22 performs correction processing on the photographed image information, and the object detection processing unit 23 performs object detection processing to detect objects. Then, among the objects in the image information, a marker is set on an object in the direction of movement of the portable communication terminal 2, preferably on the object at the end of the direction of movement of the portable communication terminal 2, or in its vicinity. For example, if the photographer moves the portable communication terminal 2 from top to bottom, a marker is set on an object in the lower direction, preferably on the object closest to the bottom, among the objects detected in the image information; and if the portable communication terminal 2 is moved from bottom to top, a marker is set on an object in the upper direction, preferably on the object closest to the top, among the objects detected in the image information, or in its vicinity.

[0215] The image capturing processor 20 then calculates the amount of movement, such as the amount of movement of the angle of view, from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the end opposite the direction in which the portable communication terminal 2 is moved, to the marker, and displays a predetermined display such as an arrow with a length proportional to the amount of movement on the display device 72 of the portable communication terminal 2. Since the direction of this arrow can identify the direction in which the portable communication terminal 2 is moved depending on the direction from the reference point to the marker, it is preferable to use the direction of the arrow as the movement direction. In other words, the image capturing processor 20 displays on the display device 72 an arrow or other display (movement display) indicating the direction and / or amount of movement of the portable communication terminal 2, superimposed on the image information captured by the image capturing device 75 of the portable communication terminal 2.

[0216] When the photographer moves the portable communication terminal 2 in the direction of the arrow, the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76 and updates the length of the displayed arrow to a length proportional to the distance traveled to the destination. As a result, as the photographer moves the orientation of the portable communication terminal 2, the length of the arrow shortens, and when the marker reaches an arbitrary point in the original image information, for example, an arbitrary point (reference point) near the end of the image information on the opposite side from the direction in which the portable communication terminal 2 is being moved, the length of the arrow becomes zero. At this time, a movement indication is displayed indicating that the portable communication terminal 2 is stopped, and the photography device 75 automatically takes a photograph. While it is preferable for the photography device 75 to take a photograph automatically, the photography processing unit 20 may also accept the photographing operation performed by the photographer.

[0217] To automatically capture images, the camera 75 is caused to capture images after it is confirmed that the person in charge of capturing images has stopped the portable communication terminal 2 and that the acceleration detected by the sensor device 76 of the portable communication terminal 2, for example, an acceleration sensor, is 0 or within a range of acceleration that can be determined as a stop. After capturing images with the camera 75, a display indicating the start of movement is displayed, for example, by calculating the amount of movement in the angle of view from the reference point to the next marker and displaying a movement such as an arrow with a length proportional to the amount of movement. This can prevent blurring and the like.

[0218] The above process is repeated until the entire display shelf has been photographed. By performing the above process, the number of images captured by the photographing processing unit 20 can be reduced, and the processing by the image selection processing unit 25 can be omitted.

[0219] FIG. 37 schematically illustrates the processing of this embodiment. FIG. 37(a) is a side view of the state in which the photographer is photographing the display shelf, and FIG. 37(b) is a front view of the state in which the photographer is photographing the display shelf. Also, p1 to p4 indicate the positions of the portable communication terminal 2 when photographing the display shelf with the photographing device 75 of the portable communication terminal 2. Note that while the photographing positions of the portable communication terminal 2 are shown shifted in FIG. 37 for ease of understanding, in reality, the positions may be the same or shifted, with only the orientation of the portable communication terminal 2 being changed.

[0220] The processing in this embodiment will be described using the flowcharts in Figures 38 and 39. Figure 38 is an example of a flowchart showing the overall processing in this embodiment, and Figure 39 is an example of a flowchart showing the photographing processing. A case will be described in which the portable communication terminal 2 is a smartphone. Similar to the case in Figure 37, a case will be described in which the photographing direction is changed from downward to upward using the portable communication terminal 2. That is, a case will be described in which the photographing direction is changed to p1, p2, p3, and p4.

[0221] First, as in Example 1, the person in charge of photographing the store's display shelves performs a predetermined operation on the store's display shelves (the object to be photographed) using the camera of the smartphone he or she owns, and the photography processing unit 20 then executes the photography process (S900).

[0222] The photographer performs a predetermined operation to launch a predetermined application software stored in the smartphone. This launch causes the photography processing unit 20 to activate the smartphone's camera. The photographer then points the smartphone's camera toward the display shelves, holds the smartphone in a fixed position so as to minimize the change in the shooting position, and begins the photography process by moving from above to below or from below to above the display shelves. That is, the photography processing unit 20 repeats the process from S1000 onwards until a photography termination condition is met, such as capturing the top of the display shelves (the space above the display shelves or the space where stock items are placed) or the bottom (the floor on which the display shelves are placed). In this case, because the smartphone's photography direction is changed from below to above, the photography termination condition is when an object is recognized at the top of the display shelves (the space above the display shelves or the space where stock items are placed).

[0223] 37, since the shooting starts from the bottom, the person in charge of shooting points the smartphone in the direction of p1, which is the shooting start position. When the shooting processing unit 20 detects that the smartphone has stopped moving to a certain extent using an acceleration sensor of the sensor device 75 (S1000), it takes the first shot (S1010). At this time, the shooting processing unit 20 may automatically take the shot using the shooting device 75, or may issue an instruction to press the shutter of the smartphone camera, causing the person in charge of shooting to take the shot.

[0224] For example, Fig. 40(a) shows an example of image information captured when the first image is captured so that the bottom part is visible. Fig. 40(a) shows the state when the display shelf is captured from position p1 so that the floor surface is visible.

[0225] The image information captured by the image capture processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 executes a correction process to correct the image to a position facing directly (S1020). This process may be a well-known keystone correction process as in Example 1, or a correction process as in Example 2. Note that if the image capture processing unit 20 has captured the image from a position facing directly, the correction processing by the correction processing unit 22 does not need to be performed. Figure 40(b) shows the image information after the correction process.

[0226] Then, the object detection processing unit 23 performs an object detection process on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing unit 22 (S1030).

[0227] That is, the object (target) area and its type (label) are detected from the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing unit 22. The object detection processing in the object detection processing unit 23 can be performed using the same method as in each of the above-mentioned embodiments. Fig. 41 shows the state in which the object area and type are detected from the image information after the correction processing in Fig. 40(b). Fig. 41 shows the state in which the area and type of each of the objects "RackBottom", "Shelf", "Faces", and "Shelf" are detected from the bottom.

[0228] The image capture processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether a capture end condition has been detected, for example, whether a predetermined object at the top or bottom of the capture direction, such as a "Space Over Rack" or "Rack Bottom," has been detected (S1040). Here, since "Space Over Rack," which indicates that the top of the display shelf has been captured, has not been detected, the image capture processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction of the smartphone capture direction, preferably on the object at the extreme end or in its vicinity (S1050). Since this object serves as a tracking object, it may also be used. In this case, since the smartphone is moved from bottom to top, a marker is set near the top of the "Shelf" object, which is located at the top. The marker and the tracking object may be the same, and all or part of the area of ​​the "Shelf" object at the top of FIG. 41 may be set as the marker.

[0229] The image processing unit 20 then calculates the amount of movement of the angle of view from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the end opposite the direction in which the smartphone is moved, to the marker (S1060), and displays a predetermined display, such as an arrow with a length proportional to the amount of movement, on the smartphone display (S1070). The direction of this arrow can be determined based on the direction from the reference point to the marker, so it is preferable to use the direction of the arrow as the movement direction. That is, the image processing unit 20 displays an arrow or other display (movement display) on the smartphone display, superimposed on the image information captured by the smartphone camera, indicating the direction and / or amount of movement of the smartphone.

[0230] FIG. 42 shows a schematic representation of this state. At this time, the smartphone display displays the pre-correction image information ( FIG. 40( b) ), so the image information may be superimposed on the image information. In this case, the amount of movement in the angle of view from the reference point to the marker is calculated using the post-correction image information, so the amount of movement in the angle of view from the reference point to the marker in the pre-correction image information may be converted into the amount of movement in the angle of view from the reference point to the marker in the pre-correction image information, and the length of the arrow may be displayed. Alternatively, since the vertical deviation between before and after correction is not significant, even if the amount of movement is calculated based on the post-correction image information, the tracking object or a marker placed near it will not deviate from the angle of view very often. Therefore, an arrow with a length corresponding to the amount of movement calculated using the post-correction image information may be superimposed on the pre-correction image information.

[0231] In either case, the image capturing processing unit 20 performs a moving display that indicates the direction and / or amount of movement of the smartphone, superimposed on the image displayed on the display of the smartphone.

[0232] When the photographer moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the distance moved to the destination (marker). An example of this state is shown in Figure 43. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0233] As the photographer moves the smartphone, the length of the arrow becomes shorter, and when an arbitrary point in the original image information, for example, an arbitrary point (reference point) near the end of the image information on the opposite side from the direction in which the smartphone is being moved, reaches the position of the original marker, the length of the arrow becomes 0. At this time, a movement display is displayed indicating that the smartphone should be stopped, and a photo is automatically taken with the smartphone camera. An example of this state is shown in Figure 44.

[0234] When this message is displayed, the person in charge of taking pictures keeps the smartphone still (S1000), and the photography processing unit 20 takes a picture again with the smartphone camera in the same manner as described above (S1010). At this time, the image information taken by the photography processing unit 20 with the smartphone camera is shown in Fig. 45(a).

[0235] Then, the image information captured by the image capture processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 executes correction processing to correct the image to a position facing directly (S1020). Figure 45(b) shows the image information after correction processing.

[0236] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing processing unit 22 (S1030). Fig. 46 shows the state in which the object areas and types have been detected for the image information after the correction processing in Fig. 45(b). Fig. 46 shows the state in which the object areas and types of each of "Shelf", "Faces", and "Shelf" have been detected from the bottom.

[0237] The photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether a "Space Over Rack" has been detected, which is the photography end condition (S1040). Here, since a "Space Over Rack" indicating that photography has been completed up to the top of the display shelf has not been detected, the photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction of moving the smartphone's photography direction, preferably on the object at the extreme end or in its vicinity (S1050). All or part of the area of ​​the object "Shelf" at the top of Figure 46 is set as the marker.

[0238] Then, the amount of movement of the angle of view from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the end on the opposite side to the direction in which the smartphone is moved, to the marker is calculated (S1060), and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the smartphone display (S1070). This state is schematically shown in Figure 47.

[0239] When the photographer moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the distance moved to the destination (marker). An example of this state is shown in Figure 48. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0240] When an arbitrary point (reference point) near the end of the image information on the opposite side to the direction in which the smartphone is being moved reaches the position of the original marker, the length of the arrow becomes 0. At that time, a movement display is displayed indicating that the smartphone is to be stopped, and a photo is automatically taken with the smartphone camera. An example of this state is shown in Figure 49.

[0241] When this message is displayed, the person in charge of taking pictures keeps the smartphone still (S1000), and the photography processing unit 20 takes a picture again with the smartphone camera in the same manner as described above (S1010). The image information taken by the photography processing unit 20 with the smartphone camera at this time is shown in Fig. 50(a).

[0242] Then, the image information captured by the image capture processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 performs correction processing to correct the image to a position facing directly (S1020). Figure 50(b) shows the image information after correction processing.

[0243] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing processing unit 22 (S1030). Fig. 51 shows the state in which the object areas and types have been detected for the image information after the correction processing in Fig. 50(b). Fig. 51 shows the state in which the areas and types of each of the objects "Shelf", "Faces", "Shelf", and "Faces" have been detected from the bottom.

[0244] The photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to determine whether a "Space Over Rack" has been detected, which is the photography end condition (S1040). In this case, since a "Space Over Rack" indicating that photography has been completed up to the top of the display shelf has not been detected, the photography processing unit 20 uses the area and type of the object detected by the object detection processing unit 23 to set a marker on an object in the direction of moving the smartphone's photography direction, preferably on the object at the extreme end or in its vicinity (S1050). All or part of the area of ​​the object "Faces" at the top of Figure 51 is set as the marker.

[0245] Then, the amount of movement of the angle of view from an arbitrary point in the image information, for example, an arbitrary point (reference point) near the end on the opposite side to the direction in which the smartphone is moved, to the marker is calculated (S1060), and a predetermined display such as an arrow with a length proportional to the amount of movement is displayed on the smartphone display (S1070). This state is schematically shown in Figure 52.

[0246] When the photographer moves the smartphone in the direction of the arrow (S1080), the photography processing unit 20 calculates the amount of movement using a sensor that detects the angle of view in the sensor device 76, and updates the length of the displayed arrow to a length proportional to the distance moved to the destination (marker). An example of this state is shown in Figure 53. The processes from S1060 to S1080 are repeated until the reference point reaches the position of the marker (S1090).

[0247] When an arbitrary point (reference point) near the end of the image information on the opposite side to the direction in which the smartphone is being moved reaches the position of the original marker, the length of the arrow becomes 0. At that time, a movement display is displayed indicating that the smartphone is to be stopped, and a photo is automatically taken with the smartphone camera. An example of this state is shown in Figure 54.

[0248] When this message is displayed, the person in charge of taking pictures keeps the smartphone still (S1000), and the photography processing unit 20 takes a picture again with the smartphone camera in the same manner as described above (S1010). At this time, the image information taken by the photography processing unit 20 with the smartphone camera is shown in Fig. 55(a).

[0249] Then, the image information captured by the image capture processing unit 20 is stored in the image information storage unit 21, and the correction processing unit 22 executes correction processing to correct the image to a position facing directly (S1020). Figure 55(b) shows the image information after correction processing.

[0250] Then, the object detection processing unit 23 performs object detection processing on the image information captured by the image capture processing unit 20 or the image information corrected by the correction processing processing unit 22 (S1030). Fig. 56 shows the state in which the object areas and types have been detected for the image information after the correction processing of Fig. 55(b). Fig. 56 shows the state in which the areas and types of each of the objects "Faces", "Shelf", "Faces", and "SpaceOverRack" have been detected from the bottom.

[0251] Then, the photography processing unit 20 determines a photography end condition, for example, whether a predetermined object has been detected, using the area and type of the object detected by the object detection processing unit 23 (S1040). In this case, the object "SpaceOverRack" indicating that photography has been completed up to the top of the display shelf has been detected, so the processing in the photography processing unit 20 ends.

[0252] After the photographing process in the photographing processing unit 20 is executed as described above, the upload processing unit 26 sends the photographed image information (image information before correction) and the parameters for performing correction on the image information output by the correction processing unit 22 to the management server 3 (S910). That is, the upload processing unit 26 sends the image information and the parameters for performing correction of each of Figures 40(a), 45(a), 50(a), and 55(a) to the management server 3.

[0253] The upload processing unit 26 may send the corrected image information, that is, the image information of FIGS. 40(b), 45(b), 50(b), and 55(b), to the management server 3.

[0254] In the upload process, the upload processing unit 26 may send a shelf structure table corresponding to each piece of image information.

[0255] Then, when the image information reception processing unit 30 of the management server 3 receives each piece of pre-correction image information and the parameters for performing the correction uploaded from the portable communication terminal 2, the correction reproduction processing unit 31 performs a correction reproduction process on the uploaded pre-correction image information using the parameters for performing the correction process (S920). Therefore, the correction reproduction process is executed to transform each piece of pre-correction image information in Figures 40(a), 45(a), 50(a), and 55(a) into post-correction image information using the parameters for performing the correction.

[0256] It should be noted that if the image information reception processing unit 30 receives the corrected image information, the correction reproduction processing in the correction reproduction processing unit 31 is not necessary.

[0257] Then, the product identification processing unit 320 in the management server 3 performs product identification processing on each corrected image information, and identifies the products displayed in the areas where products are displayed in each image information (areas labeled ``Faces'' or ``Hanging Faces'' (areas where it is determined that products are present)) (S930, S940).

[0258] That is, a product identification process is performed on each of the corrected image information of Figures 40(a), 45(a), 50(a), and 55(a), and the products displayed in the areas where the products are displayed in each of the image information are identified (S930, S940).

[0259] Similarly, the price reading processing unit 321 in the management server 3 performs a price reading process on each corrected image information, and reads the price of each price tag in the area where the price tag is located in each image information (the area labeled "Shelf" or "HangingShelf" (the price display area where it is determined that a price tag is located)) (S935, S945).

[0260] By performing the above-described processing, it is possible to take appropriate photographs when photographing in a store, and to identify products displayed on display shelves.

[0261] If the shelf structure table corresponding to each image information is also uploaded during the upload process, the structure of a single display shelf can be identified by combining the shelf structure tables based on the tracking object. Therefore, based on the results of the product identification process and price reading process for each image information, it is also possible to identify which product is displayed on which shelf and how much the price tag says on it.

[0262] By executing the above-described processing, the processing can be executed in the same manner as in the first embodiment.

[0263] When performing the object detection process, the image capture processing unit 20 may also perform an out-of-view determination process, similar to Example 1. When performing the out-of-view determination process on the corrected image information, for example, if it is determined that the left and right sides of a display shelf are out of view, a guide to step back before taking a photo may be displayed on the smartphone display.

[0264] In Examples 1 to 6, the order of processing can be changed as desired. Processing can be added or deleted without changing the spirit of the present disclosure. Furthermore, since the information processing system 1 according to one embodiment of the present disclosure can execute image correction processing with a low processing load, a portable communication terminal 2 such as a smartphone or tablet computer with low processing power is preferable as the information processing device. While this case has been described, it goes without saying that the information processing device may also be a computer or server with high processing power.

[0265] Any of the embodiments may be combined as appropriate.

[0266] By using the information processing system 1 according to one aspect of the present disclosure, it is possible to reduce the number of image information to be processed while reducing the burden of photographing. In addition, since it is possible to determine image information that is no longer suitable for photographing at the store where the photographing is performed, it is possible to re-photograph the image on the spot, thereby preventing a decrease in work efficiency.

[0267] 1: Information processing system 2: Portable communication terminal 20: Photography processing unit 21: Image information storage unit 22: Correction processing unit 23: Object detection processing unit 24: Cut-out determination processing unit 25: Image selection processing unit 26: Upload processing unit 30: Image information reception processing unit 31: Correction reproduction processing unit 32: Recognition processing unit 220: Sensor information input reception processing unit 221: Angle estimation processing unit 222: Image transformation processing unit 223: Output processing unit 320: Product identification processing unit 321: Price reading processing unit 70: Arithmetic unit 71: Storage device 72: Display device 73: Input device 74: Communication device 75: Photography device 76: Sensor device

Claims

1. An information processing system for image information photographing a display shelf, comprising: an imaging processing unit which detects the amount of movement and photographs the display shelf; a correction processing unit which performs correction processing on the image information photographed by the imaging processing unit; an object detection processing unit which detects objects appearing in the image information after the correction processing; an image cut-off determination processing unit which determines whether an object is cut off using the presence or absence of an object and / or the positional relationship of the detected object from the image information, and issues a prescribed warning if it determines that an object is cut off; an image selection processing unit which selects image information to be subjected to product identification processing and / or price reading processing from multiple pieces of image information photographed by the imaging processing unit or image information after correction processing by the correction processing unit; and a recognition processing unit which executes the product identification processing and / or the price reading processing on the selected image information.

2. An information processing system for image information photographing a display shelf, comprising: an imaging processing unit that detects the amount of movement and photographs the display shelf; a correction processing unit that performs correction processing on the image information photographed by the imaging processing unit; an object detection processing unit that detects objects that appear in the image information after the correction processing; an image selection processing unit that selects image information to be subject to product identification processing and / or price reading processing from multiple pieces of image information photographed by the imaging processing unit or image information after correction processing by the correction processing unit; and a recognition processing unit that executes the product identification processing and / or the price reading processing on the selected image information.

3. The information processing system of claim 1 or claim 2, wherein the image selection processing unit sets any object in the image information to be processed as a tracking object, and selects non-adjacent image information containing the tracking object as image information to be skipped, as image information to be subjected to the product identification processing and / or the price reading processing.

4. The information processing system of claim 3, wherein the image selection processing unit sets an object near the edge of the image information to be processed as a tracking object, and selects image information that includes the tracking object and is two or more positions away from the image information to be processed as skipped image information, as image information to be subjected to the product identification processing and / or the price reading processing.

5. The information processing system of claim 4, wherein the image selection processing unit uses position information of a tracking object in the image information to be processed and the amount of movement to determine whether the tracking object in the image information to be processed is included in the shooting range of the image information preceding the image information to be processed.

6. The information processing system according to claim 1, further comprising: an image cut-off determination processing unit that performs image cut-off determination using the presence or absence of an object detected from the image information and / or the positional relationship of the detected object.

7. The information processing system according to claim 6, wherein the cut-off determination processing unit displays a predetermined warning when it determines that cut-off image information exists.

8. An information processing system for image information photographed of a display shelf, comprising: an image capture processing unit that captures the display shelf; a correction processing unit that performs correction processing on the image information captured by the image capture processing unit; an object detection processing unit that detects objects appearing in the image information after the correction processing; an image cut-off determination processing unit that determines whether an object is cut off using the presence or absence of an object and / or the positional relationship of the detected object from the image information, and issues a prescribed warning if it determines that an object is cut off; and a recognition processing unit that performs the product identification processing and / or the price reading processing on the image information.

9. An information processing system as described in claim 6 or claim 8, wherein the cut-off determination processing unit determines cut-off image information in the vertical direction based on the presence or absence of space above a shelf or a stock product area, and an area below a shelf, from the multiple captured image information.

10. An information processing system as described in claim 6 or claim 8, wherein the cut-off determination processing unit determines cut-off image information in the left-right direction by detecting whether an area where a product is placed or a price display area is detected in the left-right center of the multiple captured image information with a size equal to or larger than a predetermined size, and whether left and right cuts or left and right structural materials are within a predetermined range from the left and right edges of the image information.

11. An information processing system according to claim 1, 2 or 8, wherein the photographing processing unit automatically performs photographing when it detects that the distance in the angle of view has moved by a predetermined value or more.

12. An information processing system for image information photographing a display shelf, comprising: an image-taking processing unit that takes an image of the display shelf; and a recognition processing unit that executes the product identification processing and / or the price reading processing on the image information, wherein the image-taking processing unit performs a correction processing on the photographed image information until a predetermined photographing end condition is satisfied, and when it detects that a reference point in the photographed image information or the corrected image information has moved to a predetermined location due to the movement of a portable communications terminal, it takes a photograph or accepts a photographing operation.

13. The information processing system of claim 12, wherein the photography processing unit performs correction processing on the captured image information until a photography end condition is satisfied using an object captured in the image information, detects an object captured in the corrected image information, sets a specified location at or near an object captured in the image information that is in the direction of movement of the portable communications terminal, and performs photography or accepts a photography operation when it detects that a reference point in the captured image information or the corrected image information has moved to the specified location due to the movement of the portable communications terminal.

14. The information processing system according to claim 13, wherein the imaging processing unit calculates the amount of movement from the reference point to the specified location, and displays the movement according to the amount of movement and / or the direction of movement.

15. The information processing system according to claim 14, wherein the photographing processing unit issues a predetermined notification when it detects that the reference point has moved to the predetermined location.

16. An information processing system as described in any one of claims 1, 2, 8 or 12, wherein the correction processing unit has: a sensor information input reception processing unit that receives input of pitch angle and roll angle information of the imaging device or information processing device and distance information to an arbitrary point on an object to be photographed; and an image transformation processing unit that performs image transformation processing on the image information, wherein the image transformation processing unit uses the distance information and the pitch angle and roll angle information to calculate a regression line of a point cloud including a plurality of the points, estimates yaw angle information of the imaging device or information processing device, and uses the estimated yaw angle information to transform the image information so that it is positioned directly facing the imaging device.

17. The information processing system 1 described in claim 16, wherein the image transformation processing unit converts distance information to an arbitrary point on the object to be photographed into three-dimensional information, rotates the point based on pitch angle and roll angle information, and estimates the yaw angle information by calculating a regression line of the point cloud when the point cloud including the point is projected onto a horizontal plane.

18. The information processing system according to claim 16, wherein the correction processing section has an output processing section that outputs correction parameters for performing image transformation processing on the image information.

19. An information processing system as described in any one of claims 1, 2, 8 or 12, wherein the recognition processing unit performs the product identification process on a product area detected by the object detection processing unit, and / or performs the price reading process on a price display area detected by the object detection processing unit.

20. The information processing system according to any one of claims 1, 2, 8 or 12, further comprising: an upload processing unit that uploads part or all of the pre-correction image information and correction parameters for correction of the image information selected by the image selection processing unit; and a correction reproduction processing unit that reproduces the correction using part or all of the uploaded pre-correction image information and the correction parameters, and the recognition processing unit performs product identification processing and / or price reading processing on part or all of the corrected image information reproduced by the correction reproduction processing unit.

21. An information processing program that causes a computer to function as: an imaging processing unit that detects the amount of movement and photographs a display shelf; a correction processing unit that performs correction processing on image information captured by the imaging processing unit; an object detection processing unit that detects objects appearing in the image information after the correction processing; an image cut-off determination processing unit that determines whether an object is detected from the image information and / or the positional relationship of the detected object, and issues a prescribed warning if it determines that the object is cut off; an image selection processing unit that selects image information to be subjected to product identification processing and / or price reading processing from multiple pieces of image information captured by the imaging processing unit or image information after correction processing by the correction processing unit; and a recognition processing unit that executes the product identification processing and / or the price reading processing on the selected image information.

22. An information processing program that causes a computer to function as: an imaging processing unit that detects the amount of movement and photographs a display shelf; a correction processing unit that performs correction processing on image information captured by the imaging processing unit; an object detection processing unit that detects objects that appear in the image information after the correction processing; an image selection processing unit that selects image information to be subject to product identification processing and / or price reading processing from multiple pieces of image information captured by the imaging processing unit or image information after correction processing by the correction processing unit; and a recognition processing unit that executes the product identification processing and / or the price reading processing on the selected image information.

23. An information processing program that causes a computer to function as: an imaging processing unit that photographs a display shelf; a correction processing unit that performs correction processing on image information captured by the imaging processing unit; an object detection processing unit that detects objects appearing in the image information after the correction processing; a cut-off determination processing unit that determines whether an object is cut off using the presence or absence of an object and / or the positional relationship of the detected object from the image information, and issues a specified warning if it determines that an object is cut off; and a recognition processing unit that executes the product identification processing and / or the price reading processing on the image information.

24. An information processing program that causes a computer to function as an image capture processing unit that captures images of display shelves, and a recognition processing unit that executes the product identification process and / or the price reading process on the image information, wherein the image capture processing unit performs correction processing on the captured image information until a predetermined image capture end condition is satisfied, and performs an image capture or accepts an image capture operation when it detects that a reference point in the captured image information or the corrected image information has moved to a predetermined location due to the movement of a portable communications terminal.

25. A portable communications terminal for use in an information processing system that photographs a display shelf and recognizes the products and / or prices displayed in the image information, comprising: an imaging processing unit that detects the amount of movement by an imaging device of the portable communications terminal and photographs the display shelf; a correction processing unit that performs correction processing on the image information photographed by the imaging processing unit; an object detection processing unit that detects objects that appear in the image information after the correction processing; an image selection processing unit that selects image information to be subject to product identification processing and / or price reading processing from multiple pieces of image information photographed by the imaging processing unit or image information after correction processing by the correction processing unit; an upload processing unit that uploads some or all of the selected image information in order to execute product identification processing and / or price reading processing using the image information selected by the image selection processing unit; and an information processing program.

26. An information processing program that causes a portable communications terminal used in an information processing system that photographs display shelves and recognizes the products and / or prices displayed in the image information to function as: an imaging processing unit that photographs the display shelves; and an upload processing unit that uploads part or all of the photographed image information or corrected image information in order to execute product identification processing and / or price reading processing using the image information, wherein the imaging processing unit performs correction processing on the photographed image information until a specified imaging end condition is satisfied, and performs an imaging operation or accepts an imaging operation when it detects that a reference point in the photographed image information or corrected image information has moved to a specified location due to the movement of the portable communications terminal.

Images