Information processing system

Abstract

To provide an information processing system that detects a region of a display shelf to be processed during an identification / price-tag-reading process performed on a commodity displayed on the display shelf.SOLUTION: An information processing system for image information for imaging a display shelf includes: an image information reception processing unit that receives input of image information in which a display shelf to be processed appears; and a directly facing processing unit which executes directly facing processing to modify the input image information in which the display shelf to be processed appears, so as to be located in a directly facing position. The directly facing processing unit estimates a positional relationship between a target surface and an information processing terminal on a three-dimensional space, using sensor information detected by a sensor device of the information processing terminal that captured the image of the display shelf, to perform the directly facing processing.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing system that detects an area of a display shelf to be processed when identifying products displayed on the display shelf or reading price tags.

Background Art

[0002] In various stores such as convenience stores and supermarkets, it is common to sell products by placing them on display shelves. Therefore, by arranging multiple products on the display shelf, even if one product is purchased, the same product can be purchased by others. Also, products arranged in a large number in a prominent position are noticed, and a competitive relationship of superiority or inferiority occurs, where they sell more than other products. Therefore, managing where and how many products are displayed on the display shelf and the prices of those products is important in terms of the sales strategy of the products.

[0003] Therefore, the display shelf is photographed with a photographing device such as a camera, and by automatically recognizing the objects shown in the image information, the display position and number of products, and the prices described on the products and price tags (price cards) are grasped. Therefore, there are the technologies disclosed in Patent Document 1 and Patent Document 2 below.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In many stores, products are displayed by aligning product types in units of display shelves. Therefore, when performing identification processing of products displayed on the display shelves or price tag reading processing, it is reasonable to perform these operations in units of display shelves.

[0006] When photographing products displayed on a display shelf with a photographing device, the photographing is performed so that the left and right ends of one display shelf are within the angle of view. However, if one tries to fit the display shelf exactly within the angle of view, it will take time to adjust the photographing angle when photographing the display shelf with the photographing device. Therefore, in practice, it is common to perform the photographing with a margin on the left and right of the angle of view.

[0007] Since the photographing is performed with a margin on the left and right of the angle of view, the captured image information will originally include, in addition to the display shelf that is the object of photographing, another display shelf located to the left and / or right of that display shelf and the products displayed thereon. Therefore, when attempting to perform identification processing of products displayed in units of display shelves or price tag reading processing, the display shelves on the left and right adjacent to the original processing target display shelf and the products displayed thereon need to be excluded from the processing target.

[0008] However, starting from the above-mentioned patent documents and other prior arts, it has not been possible to accurately exclude the display shelves on the left and right adjacent to each other and the products displayed thereon from the processing target. In some cases, humans may manually set the area to be processed from the captured image.

Means for Solving the Problem

[0009] In view of the above problems, the inventor has invented an information processing system that detects the area of the display shelf that is the processing target during the identification / price tag reading processing of products displayed on the display shelf.

[0010] The first invention is an information processing system for image information of photographing a display shelf. The information processing system includes an image information reception processing unit that receives an input of image information in which the display shelf that is the processing target appears, and the input received processing target The display shelf in the image information is for the imaging device or information processing terminal that has captured the display shelfso as to be in a facing position , the image information to be processed a normalization processing unit that executes a normalization process to deform so as to be in a facing position, and has, wherein the normalization processing unit the information processing terminal Using the sensor information detected by the sensor device of, an information processing system that estimates the positional relationship between the information processing terminal and the target surface in three-dimensional space and performs the normalization process.

[0011] A second invention is an information processing system for image information obtained by photographing a display shelf, wherein the information processing system includes an image information reception processing unit that receives an input of image information in which a display shelf to be processed is captured, and the input is received for the processing target The display shelf in the image information is for the imaging device or information processing terminal that has captured the display shelf so as to be in a facing position , the image information to be processed a normalization processing unit that executes a normalization process to deform so as to be in a facing position, and has, wherein the normalization processing unit executes the normalization process using the distance information to the photographed object acquired from a sensor device that measures the distance to the photographed object, It is an information processing system.

[0012] When photographing a display shelf in a store, it may not be possible to secure the distance between the display shelf to be photographed and the photographing device. In addition, the photographing is often performed at an elevation angle or a depression angle, and it is often not possible to photograph from a facing position. If such image information is processed as it is, it will lead to misrecognition. Therefore, it is preferable to correct the image information photographed by the normalization processing unit so as to be in a facing position as if it was photographed from a facing position.

[0013] By configuring as in these inventions, a correction process with a small processing load becomes possible. Therefore, even a portable communication terminal with low capabilities such as a smartphone can perform a correction process.

[0014] In the above invention, the normalization processing unit can be configured as an information processing system that converts the distance information to the photographed object acquired from a sensor device that measures the distance to the photographed object into three-dimensional information and executes the normalization process.

[0015] In the above invention, the uprighting processing unit receives the input of the information on the pitch angle and roll angle of the imaging device that has photographed the display shelf or the information processing terminal, calculates the regression line of the point group including a plurality of points by using the distance information, the information on the pitch angle, and the information on the roll angle, estimates the information on the yaw angle of the imaging device or the information processing terminal, and deforms the image information so as to be in a position facing the imaging device by using the estimated information on the yaw angle, and can be configured as an information processing system.

[0016] In the above invention, the uprighting processing unit can be configured as an information processing system that converts the distance information to an arbitrary point in the imaging object into three-dimensional information, rotates the point based on the information on the pitch angle and roll angle, and estimates the information on the yaw angle by calculating the regression line of the point group when the point group including the point is projected onto a horizontal plane.

[0017] By configuring as these inventions, correction processing with a not-so-large processing load becomes possible. Therefore, correction processing is also possible with a portable communication terminal with low capabilities such as a smartphone.

[0018] In the above invention, the information processing system can be configured as an information processing system having a recognition processing unit that recognizes the product display area and / or the price tag area on the display shelf shown in the uprighted image information by using a machine learning network generated by using annotation data annotated with the product display area and / or the price tag area on the display shelf.

[0019] By using the present invention, when identifying the products displayed on the display shelf / reading the price tags, the area of the display shelf to be processed can be detected.

[0020] In the above invention, the information processing system may be configured to include a learning processing unit that generates a machine learning network, and a detection processing unit that performs detection processing of a predetermined area from image information using the network. The learning processing unit may include an annotation data reception processing unit that receives input of annotation data annotated with a product display area and / or a price tag area in the display shelf, and a learning processing unit that performs learning processing using the received annotation data and generates a machine learning network.

[0021] The information processing system of the first invention can be realized by causing a computer to read and execute the information processing program of the present invention. That is, the computer functions as an image information reception processing unit that receives input of image information showing a display shelf to be processed, and an ortho-positioning processing unit that executes ortho-positioning processing to deform so as to be in a directly facing position with respect to the received input. The ortho-positioning processing unit estimates the positional relationship between the information processing terminal and the target surface in the three-dimensional space using the sensor information detected by the sensor device and performs the ortho-positioning processing. The display shelf in the image information is for the imaging device or information processing terminal that has captured the display shelf so as to be in a directly facing position , the image information to be processed An information processing program that causes the computer to function as an ortho-positioning processing unit that executes ortho-positioning processing to deform so as to be in a directly facing position with respect to the received input. The ortho-positioning processing unit performs the ortho-positioning processing using the distance information to the photographed object acquired from a sensor device that measures the distance to the photographed object. the information processing terminal using the sensor information detected by the sensor device, estimates the positional relationship between the information processing terminal and the target surface in the three-dimensional space, and performs the ortho-positioning processing.

[0022] The information processing system of the second invention can be realized by causing a computer to read and execute the information processing program of the present invention. That is, the computer functions as an image information reception processing unit that receives input of image information showing a display shelf to be processed, and an ortho-positioning processing unit that executes ortho-positioning processing to deform so as to be in a directly facing position with respect to the received input. The ortho-positioning processing unit performs the ortho-positioning processing using the distance information to the photographed object acquired from a sensor device that measures the distance to the photographed object. The display shelf in the image information is for the imaging device or information processing terminal that has captured the display shelf so as to be in a directly facing position , the image information to be processed An information processing program that causes the computer to function as an ortho-positioning processing unit that executes ortho-positioning processing to deform so as to be in a directly facing position with respect to the received input. The ortho-positioning processing unit performs the ortho-positioning processing using the distance information to the photographed object acquired from a sensor device that measures the distance to the photographed object.

Advantages of the Invention

[0023] By using the information processing system of the present invention, it is possible to detect the area of the display shelf to be processed during the identification / price tag reading process of the products displayed on the display shelf.

Brief Description of the Drawings

[0024]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Mode for Carrying Out the Invention

[0025] A block diagram of an example of the overall processing function of the information processing system 1 of the present invention is shown in FIG. 1. The information processing system 1 functions in the information processing terminal 2. As the information processing terminal 2, various computers such as a portable communication terminal such as a smartphone or a tablet computer, a laptop computer, a desktop computer, or a server may be used. The information processing terminal 2 may have any appearance or name as long as it has the functions necessary for the processing of the information processing system 1 of the present invention.

[0026] An example of the hardware configuration of the information processing terminal 2 in the information processing system 1 is schematically shown in FIG. 2. The information processing terminal 2 includes an arithmetic unit 70 such as a CPU that executes arithmetic processing of programs, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 through which information can be input, and a communication device 74 that transmits and receives the processing results of the arithmetic unit 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0027] When the information processing terminal 2 is provided with a touch panel display, the display device 72 and the input device 73 may be integrally configured. The touch panel display is often used in information processing terminals 2 such as tablet computers and smartphones, for example, but is not limited thereto.

[0028] The touch panel display is a device in which the functions of the display device 72 and the input device 73 are integrated in that input can be directly performed on the display by a predetermined input device (such as a pen for a touch panel) or a finger.

[0029] Each means in the present invention only has its functions logically distinguished, and may physically or practically form the same area. The processing in each means of the present invention can also have its processing order appropriately changed. Also, a part of the processing may be omitted.

[0030] In the information processing system 1, it is not necessary to execute the processing on one information processing terminal 2, and the functions may be distributed and executed on a plurality of information processing terminals 2. For example, the processing of the learning processing unit 20 and the processing of the detection processing unit 21 described later may be executed on different information processing terminals 2. Either or both may be executed on a cloud server.

[0031] The information processing terminal 2 has a learning processing unit 20 and a detection processing unit 21.

[0032] The learning processing unit 20 receives the input of image information in which a display shelf is shown, and executes a process of learning a network (learning model) for detecting the area of the display shelf to be processed during the process of identifying the products displayed on the display shelf and reading the price tags. That is, the learning processing unit 20 generates a network (learning model) for executing a process of detecting a region to be processed from the display shelf shown in the image information by using deep learning such as machine learning (deep learning) on the image information in the detection processing unit 21 described later.

[0033] The learning processing unit 20 includes an annotation data reception processing unit 201 and a network generation processing unit 202.

[0034] The annotation data reception processing unit 201 receives the input of annotation data for use in the learning of the network. As the annotation data, for example, image information in which a display shelf displayed in a store is shown, and image information in which elements constituting the display shelf, such as a product display area and a price tag (price card) area, are set as annotations can be cited. For the product display area, price tag area, etc. in the annotation data, it is preferable to use the image information in which the entire area is shown as the annotation data, and the positions of the left and right ends of the product display area and the price tag area are set as accurately as possible. The annotations in the annotation data are not limited to those described above, and any element constituting the display shelf can be set as an annotation.

[0035] The product display area and price tag area set as annotations can be set by any method such as the coordinates of four points of a rectangle constituting the area, the coordinates of two points of the upper left and lower right, the upper right and lower left, the coordinates of one point and the lengths in the x direction and y direction from that point. Also, the image information used as the annotation data is preferably image information that has been processed such as trapezoidal correction processing so that the image information is taken from a directly facing position and the annotation area is set.

[0036] The network generation processing unit 202 uses the annotation data received by the annotation data reception processing unit 201 to execute the learning process of deep learning by a known learning method in deep learning, and generates a network (learning model). The network generated by the network generation processing unit 202 is used to execute the recognition by the recognition processing unit 303 in the detection processing unit 21 described later.

[0037] The detection processing unit 21 uses the network generated by the learning processing unit 20 to detect the area to be processed during the identification / price tag reading process of the products displayed on the display shelf from the image information to be processed. The detection processing unit 21 includes an image information reception processing unit 301, an alignment processing unit 302, a recognition processing unit 303, a correction processing unit 304, and an output processing unit 305.

[0038] The image information reception processing unit 301 receives the input of the image information to be processed. The image information at this time may be one piece of image information showing the entire display shelf, or may be image information obtained by photographing the display shelf with a plurality of pieces of image information and connecting them into one piece of image information to form the entire display shelf. Furthermore, it may be image information showing a part of the display shelf.

[0039] The alignment processing unit 302 performs correction processing (alignment processing) on the image information to be processed received by the image information reception processing unit 301 so that it is in a facing position. As the alignment processing unit 302, any processing may be used as long as it can correct the image information to a facing position. For example, a known trapezoidal correction processing may be used.

[0040] When photographing the display shelf of a store, due to circumstances such as another display shelf being located behind the photographer and the passage being narrow, it may not be possible to ensure the distance between the display shelf to be photographed and the photographing device. In addition, since the display shelf is located above or below the line of sight of the photographer, the photographing is often at an elevation angle or a depression angle. Therefore, it is often not photographed from a facing position. The image information photographed by the alignment processing unit 302 is corrected so that it is in a facing position as if it were photographed from a facing position.

[0041] Note that the uprighting processing unit 302 may generate image information obtained by performing uprighting processing on the image information to be processed, or may output parameters (correction parameters) for deforming the image information so as to be in a position facing the front. For example, parameters (parameters used for projective transformation (correction parameters)) for generating uprighted image information may be output so as to perform projective transformation on the image information to be in a position facing the front.

[0042] If the image information received by the image information reception processing unit 301 is image information captured from a position facing the front, the processing of the uprighting processing unit 302 may not be performed.

[0043] The recognition processing unit 303 recognizes, from the image information to be processed that has been uprighted by the uprighting processing unit 302, an area for performing identification / price tag reading processing of the products displayed on the display shelf. As this recognition processing, for the image information to be processed that has been uprighted by the uprighting processing unit 302, processing for recognizing each of the above areas can be executed using deep learning (deep neural network). In this case, for a network (learning model) in which the weight coefficients between the neurons of each layer of a neural network having a large number of intermediate layers are optimized, the image information uprighted by the uprighting processing unit 302 is input, and an area for performing identification / price tag reading processing of the products from the display shelf shown in the image information is recognized. As the network (learning model), the network generated by the learning processing unit 20 can be used.

[0044] The correction processing unit 304 executes processing for correcting the position when the left and right end portions of the area for performing identification / price tag reading processing of the products from the display shelf recognized by the recognition processing unit 303 are displaced.

[0045] As the correction processing unit 304, for example, the following processing can be used.

[0046] As the first correction process, it is a method of detecting abnormal values at the left and right ends of the area and adjusting the area with the abnormal values to the positions of the left and right ends of other areas. That is, most of the areas recognized by the recognition processing unit 303 are correctly detected. Therefore, when determining the positions of the left and right ends of the recognized area and determining that there are abnormal values (for example, the positions of the left and right ends that deviate from the average value or median value of the recognized area by a predetermined size or ratio), the positions of the left and right ends of the area with the abnormal values are changed to the average value, median value, etc. of other recognized areas.

[0047] As the second correction process, it is a method of detecting objects near the ends of the display shelves. That is, various objects such as the column members of the display shelves, the novel-shaped displays for guiding products, the joints of the display shelves, and the support members for supporting products may be attached near the ends of the display shelves. An example of an object is shown in FIGS. 5 and 6. FIG. 5 shows the case of a column member as an object, and FIG. 6 shows the case of a joint of a display shelf as an object. These objects are learned in the learning processing unit 20 as annotation data, and this is the method of detecting them.

[0048] Since the left and right ends of the display shelves are common to each shelf level, by detecting the objects at the left and right ends, the positions of the left and right ends of the area recognized by the recognition processing unit 303 can be corrected by setting the positions of the left and right ends of the area recognized by the recognition processing unit 303 as the positions of the objects. For example, the positions of the left and right ends of the area recognized by the recognition processing unit 303 are corrected by calculating the average value and median value of the positions of the left and right ends of the area recognized by the recognition processing unit 303 and the positions of the recognized objects.

[0049] The output processing unit 305 outputs the area recognized by the recognition processing unit 303 or the area after the correction process by the correction processing unit 304 as a detection result.

Example

[0050] Next, an example of the process using the information processing system 1 of the present invention will be described with reference to the flowcharts of FIGS. 3 and 4.

[0051] First, the learning process will be described using the flowchart of FIG. 3.

[0052] Image information showing a display shelf displayed in a store and image information in which elements constituting the display shelf, for example, a product display area where products are displayed and a price tag area where price tags are placed, are set as annotations are input into the information processing terminal 2 as annotation data.

[0053] The input annotation data is received by the annotation data reception processing unit 201 in the learning processing unit 20 (S100). As the annotation data, it is preferable to receive the number of inputs required for learning.

[0054] Then, the network generation processing unit 202 uses the annotation data received by the annotation data reception processing unit 201 to generate a network (learning model) using a known learning processing method for deep learning (S110).

[0055] By executing the above-described processing, a network (learning model) can be generated.

[0056] Next, the detection process will be described using the flowchart of FIG. 4.

[0057] Image information for executing the identification process of products displayed on the display shelf / the reading process of price tags is input into the information processing terminal 2 as image information to be processed. The image information to be processed is received by the image information reception processing unit 301 of the detection processing unit 21 (S200).

[0058] Then, the normalization processing unit 302 executes normalization processing on the input image information to be processed (S210). An example of the image information after the normalization processing is shown in FIG. 7.

[0059] The upright image information to be processed is input as input image information to the recognition processing unit 303, and using the network (learning model) generated by the learning processing unit 20, recognition processing of elements constituting a display shelf such as a product display area and a price tag area is performed by deep learning (deep neural network) (S220). An example of this recognition result is shown in FIG. 8. In FIG. 8, the product display area and the price tag area are displayed in one image information, but the product display area and the price tag area may be output to different image information respectively. FIG. 9 shows the output of only the product display area, and FIG. 10 shows the output of only the price tag area.

[0060] Then, correction processing is performed on the recognition result by the recognition processing unit 303 using first correction processing, second correction processing, etc. by the correction processing unit 304 (S230). For example, in the recognition results in S220 (FIG. 8, FIG. 9), since the position of the left end of the fifth area from the top of the product display area ("5Faces") is shifted, the position of the left end is corrected using first correction processing, second correction processing, etc. The results after the correction processing are shown in FIGS. 11 and 12.

[0061] The output processing unit 305 outputs the result after the correction processing as a detection result as described above. That is, the results of FIG. 11, or FIGS. 12 and 10 are output as the detection result. Note that the output processing unit 305 may output not only the image information but also the position coordinates (coordinate information, or coordinate information and length) of the product display area and the price tag area.

[0062] For the product display area and the price tag area in the image information of the display shelf copied as output by the output processing unit 305 as described above, product identification processing, price tag and price reading processing are performed by a known method.

Example

[0063] In the above-described Example 1, the case where various known ortho-positioning processing techniques capable of executing processing such that the positions face each other can be used as the ortho-positioning processing in the ortho-positioning processing unit 302 was described. However, the case where ortho-positioning processing with a reduced processing load can be realized can be described. As a result, even in the information processing terminal 2 with low processing power such as a smartphone, the ortho-positioning processing can be executed with a reduced load.

[0064] An example of the configuration of the ortho-positioning processing unit 302 in this embodiment is shown in FIG. 13, and an example of the ortho-positioning processing in the ortho-positioning processing unit 302 is shown in the flowchart of FIG. 14. Further, when performing the ortho-positioning processing of the present invention, the information processing terminal 2 is, for example, a portable communication terminal such as a smartphone, and has a photographing device (not shown) such as a camera and various sensor devices (not shown).

[0065] The photographing device may be any device that photographs image information using visible light or the like.

[0066] Examples of the sensor device include a sensor that detects the angle of view being photographed by the photographing device, a sensor that detects the pitch angle and roll angle of the information processing terminal 2 or the photographing device, a distance measuring sensor that measures the distance from the photographing device to the object to be photographed, etc., and it is preferable to include some or all of them.

[0067] In this embodiment, the ortho-positioning processing unit 302 regards the front surface of the object to be photographed as a vertical rectangle (including a square) in a three-dimensional space (this surface is called the target surface), and uses the sensor information detected by the sensor device of the information processing terminal 2 to estimate the positional relationship between the information processing terminal 2 and the target surface in the three-dimensional space, thereby performing ortho-positioning processing. FIG. 15 shows an example of the image information (image information before correction) photographed by the photographing device.

[0068] The ortho-positioning processing unit 302 includes a sensor information input reception processing unit 3021, an angle estimation processing unit 3022, an image deformation processing unit 3023, and an ortho-positioned image output processing unit 3024.

[0069] The sensor information input reception processing unit 3021 receives inputs of information such as the pitch angle and roll angle of the information processing terminal 2 or the imaging device from a sensor that detects them from the sensor device. Further, it receives an input of distance information between the imaging device and the imaging object from a sensor (distance measuring sensor) that measures the distance to the imaging object. As the distance information, distance information to each point from the imaging device to the imaging object is acquired.

[0070] The angle estimation processing unit 3022 converts the distance information to each point of the imaging object, which is input by the sensor information input reception processing unit 3021, into three-dimensional information, and rotates this point cloud based on the roll angle and pitch angle. Since the target plane is vertical, as a result of the rotation, it becomes a plane perpendicular to the xz plane, roughly as shown in FIG. 16. Then, by calculating the regression line of these point clouds in the xz plane, the perpendicular line drawn from the imaging device to the line is estimated as the yaw angle.

[0071] The image deformation processing unit 3023 uses the yaw angle estimated by the angle estimation processing unit 3022 to deform the image information captured by the imaging device so that it is in a position directly facing the imaging device. The shape of the quadrilateral of the target plane represents the angle of view of the imaging device in three-dimensional space. Assuming four vectors with the imaging device as the end point, after rotating them based on the roll angle and pitch angle input by the sensor information input reception processing unit 3021, four intersection points with a vertical plane tilted left and right by the yaw angle estimated by the angle estimation processing unit 3022 are calculated. Then, by performing a projective transformation process such as OpenCV using the calculated four intersection points on the image information captured by the imaging device, image deformation processing is performed, and orthorectification processing can be executed on the image information from the directly facing position. FIG. 17 shows an example of image information after orthorectification processing on the image information of FIG. 15.

[0072] The orthorectified image output processing unit 3024 outputs the image information deformed by the image deformation processing unit 3023. For example, it outputs the image information corrected as shown in FIG. 17. Further, the orthorectified image output processing unit 3024 may output parameters for performing orthorectification, that is, parameters for performing projective transformation processing.

[0073] Next, the normalization process of the normalization processing unit 302 in the second embodiment will be described with reference to the flowchart of FIG. 14. Also, the case where the information processing terminal 2 is a smartphone will be described.

[0074] First, a photographer who takes a picture of the store's display shelf uses the camera of a smartphone that holds the store's display shelf (object to be photographed) to take a picture. The image information taken with the smartphone camera is stored (S300). An example of this image information is shown in FIG. 15. The image information in FIG. 15 is taken from the lower diagonal direction of the display shelf upward (elevation angle), and is also in a state of being slightly rotated to the left (pitch angle is 18.8 degrees upward, roll angle is 4.6 degrees counterclockwise).

[0075] In addition, the sensor information input reception processing unit 3021 receives the input of the pitch angle, roll angle information of the smartphone or camera at the time of taking the image information by the imaging device, which is measured by the sensor device, and the distance information to each point of the object to be photographed (S300).

[0076] As for the distance information, an arbitrary number of distance information, for example, about several tens to one hundred locations, may be acquired with respect to the object to be photographed. Note that the number of locations for acquiring the distance information may be any number as long as sufficient accuracy can be obtained.

[0077] The angle estimation processing unit 3022 plots the distance information input by the sensor information input reception processing unit 3021 in a three-dimensional space and converts the distance information into a point cloud in the three-dimensional space (S310). Then, the angle estimation processing unit 3022 rotates this point cloud using the pitch angle and roll angle information input by the sensor information input reception processing unit 3021 (S320).

[0078] In order to generate orthographic image information (trapezoid correction image information) in which the products displayed on the display shelves on the target surface are upright and the shelf levels are horizontal, that is, in order to orthographically position the target surface so that it faces the position directly opposite to the image information captured by the imaging device, it is necessary to calculate how much the target surface is tilted from the direction facing the camera. Therefore, the point cloud after rotation in S320 is mapped onto the horizontal plane (xz plane), and a regression line is calculated using an algorithm for robust estimation such as RANSAC (S330). FIG. 18 is a front view (xy plane of FIG. 16) of the state where the point cloud after rotation in S320 is mapped onto the vertical plane, and FIG. 19 is a top view (xz plane of FIG. 16) of the state where the point cloud after rotation in S320 is mapped onto the horizontal plane. Also, FIG. 20 schematically shows the processing in the angle estimation processing unit 3022.

[0079] Then, the angle (yaw angle) indicating how much the target surface is tilted from the optical axis of the camera can be calculated from this regression line. Specifically, assuming that the slope of the regression line of the point cloud is a, by calculating Equation 1, the slope a of the regression line is converted into the yaw angle (S340). That is, the inverse tangent is calculated with the slope a as the argument. (Equation 1) Yaw angle = atan(a)

[0080] Since the angle estimation processing unit 3022 can estimate the yaw angle by the above processing, the image deformation processing unit 3023 deforms the image information captured by the imaging device so as to be in a position directly facing the imaging device using the estimated yaw angle. This processing is schematically shown in FIG. 21.

[0081] Since the angle estimation processing unit 3022 has estimated the yaw angle, the positional relationship in the three-dimensional space between the camera and the target surface has been determined. Therefore, the image deformation processing unit 3023 sets four straight lines (vectors) connecting the four vertices formed by the horizontal plane, the camera position, and the viewing angle of the camera in the three-dimensional space (S350). Then, these four straight lines (vectors) are rotated using the roll angle and pitch angle input by the sensor information input reception processing unit 3021 and the yaw angle estimated by the angle estimation processing unit 3022 (S360).

[0082] The image deformation processing unit 3023 calculates, as a target quadrilateral, the quadrilateral of the cross-section obtained by cutting the four straight lines rotated in S360 by the target plane (S370). The image deformation processing unit 3023 performs projective transformation on the image information captured by the imaging device with respect to this target quadrilateral to generate image information after ortho-rectification (S380). The image information after the ortho-rectification process in the ortho-rectification processing unit 302 with respect to the image information in FIG. 15 is the image information in FIG. 17.

[0083] By executing the above processing, the processing of the ortho-rectification processing unit 302 can be executed.

[0084] Then, the ortho-rectification image output processing unit 3024 outputs the image information deformed by the image deformation processing unit 3023. Further, the ortho-rectification image output processing unit 3024 may output, as parameters used at the time of ortho-rectification, the parameters for performing projective transformation with respect to the target quadrilateral. By outputting the parameters, the smartphone can, instead of the image information after ortho-rectification, send the image information before ortho-rectification and the parameters, and a process of reproducing the ortho-rectification process using the image information before ortho-rectification and the parameters at a predetermined server becomes possible, and the amount of data to be uploaded can be reduced.

[0085] In addition to the processing of the above-described ortho-rectification processing unit 302, the processing can also be performed as follows.

[0086] That is, when the object to be photographed has many irregularities like a display shelf on which products are displayed, since the correlation in the horizontal direction of the irregularities of the object to be photographed is extremely strong compared to the vertical direction, if the left - right direction of the photographing device is slightly deviated from the horizontal direction of the display shelf, a "step - like difference" where the distance information of objects arranged horizontally is close up to a certain point and far from that point onward is likely to occur. And if several such sequences are mixed into the acquired distance information, the estimated result will be inclined with respect to the original target surface. Fig. 22 schematically shows this. Fig. 22(a) is image information obtained by photographing a display shelf with many irregularities, for example, a display shelf on which products are displayed, as the object to be photographed. Fig. 22(a) shows that the positions (points) for acquiring distance information may be the positions of products located in the foreground, the positions of the back shelf boards without products, etc. In that case, the positional relationship of the point cloud is as shown in Fig. 22(b), and an inclined estimated result is likely to occur.

[0087] Therefore, in the alignment process in the second embodiment, further, the target surface may be divided into matrices of a predetermined size, and distance information of points randomly scattered pseudo - randomly in each matrix may be acquired. For example, observation points (one or more angles of up, down, left, or right) with randomness are set by shifting the positions from the intersections of each matrix, the distance to the observation points is measured along this, and the measured distance information is converted into a point cloud in three - dimensional space along the set angles, so that the regularity of the shape of the object to be photographed does not cause a bias in the measured distance information, and the accuracy of the estimated result can be improved. As a result, when converting the distance information into a point cloud in three - dimensional space in S310, it becomes possible to randomly select a point cloud from the matrix among the point clouds obtained by plotting the distance information. Fig. 23 schematically shows this.

[0088] The accuracy of the estimated result can be improved by the above - mentioned processing.

[0089] Furthermore, the alignment processing unit 302 may perform processing as follows.

[0090] That is, when the object to be photographed is a display shelf, usually, the photographer stands near the center of the display shelf to be photographed and takes a picture so that the entire left and right sides of the display shelf are included. In that case, usually, a part of the adjacent display shelf may be reflected in the vicinity of the left and right ends of the photographed image information. An example of the image information showing this is Fig. 24. In the case of such image information, when the left and right display shelves protrude forward or are recessed backward from the display shelf that is the original object to be photographed, the regression line in S330 may be inclined. When the photographed image information is as shown in Fig. 24(a), the display shelf reflected on the right side of the original display shelf to be photographed protrudes forward. Therefore, when the distance information is plotted in the three-dimensional space, abnormal values occur on the right side as shown in the top view (xz plane) of Fig. 24(b). If these point clouds are directly used as the processing target, it will affect the inclination of the regression line.

[0091] Therefore, in the alignment process in the alignment processing unit 302 of the second embodiment, when calculating the regression line using a process such as RANSAC, the following process may be executed.

[0092] First, using a process such as RANSAC, a predetermined number of points are randomly selected from the point cloud after rotation in S320 to obtain a regression line. Then, for all the point clouds, the number of points whose error from the regression line falls within the threshold is used as an evaluation function, and this operation is repeated multiple times, and the regression line with the largest number of points is selected as the final regression line. For example, when obtaining a regression line from a point cloud of 100 points, a regression line is obtained by selecting 3 points as the predetermined number of points, and sufficient accuracy can be obtained by repeating the process 500 times.

[0093] Also, regarding this evaluation function, when selecting points on the outer edge, the weight can be changed to be reduced so as to reduce the influence of the unevenness of the display shelf reflected in the vicinity of the left and right ends. As a change in the weighting, for example, for points on the outer edge, instead of counting as 1, a method such as counting as 0.5 can be used.

[0094] More specifically, for example, as shown in FIG. 25, the acquisition interval of the point group is equally divided into a predetermined number, for example, 4 in the x-axis direction, and in the outer intervals, as the position approaches the outer edge, the weighting of the number of points is changed.

[0095] The ortho-processing in the ortho-processing unit 302 in the second embodiment can be similarly applied not only to the image information captured at the elevation angle or the depression angle but also to the image information captured from the left-right direction.

[0096] In the first and second embodiments, the order of the processes can be arbitrarily changed. Also, additions and deletions of processes can be made to the extent that the gist of the present invention is not changed. Furthermore, since the information processing system 1 of the second embodiment has a low processing load and can execute the ortho-processing, as the information processing terminal 2, it can be applied to a smartphone or a tablet computer with low processing power, etc., but needless to say, it may also be executed on a computer or a server with high processing power as the information processing terminal 2.

Industrial Applicability

[0097] By using the information processing system 1 of the present invention, it is possible to detect the area of the display shelf to be processed during the identification / price tag reading process of the products displayed on the display shelf.

Explanation of Signs

[0098] 1: Information processing system 2: Information processing terminal 20: Learning processing unit 21: Detection processing unit 70: Arithmetic unit 71: Storage device 72: Display device 73: Input device 74: Communication device 201: Annotation data reception processing unit 202: Network generation processing unit 301: Image information reception processing unit 302: Ortho-processing unit 303: Recognition processing unit 304: Correction Processing Unit 305: Output Processing Unit 3021: Sensor Information Input Reception Processing Unit 3022: Angle Estimation Processing Unit 3023: Image Deformation Processing Unit 3024: Rectified Image Output Processing Unit

Claims

1. An information processing system for image information obtained by photographing a display shelf, wherein the information processing system comprises an image information reception processing unit configured to receive input of image information in which a display shelf to be processed is imaged, and an ortho-positioning processing unit configured to perform ortho-positioning processing for deforming the image information to be processed such that the display shelf in the input image information to be processed is positioned directly facing the imaging device or the information processing terminal that has photographed the display shelf, and the ortho-positioning processing unit estimates the positional relationship between the information processing terminal and the target plane in a three-dimensional space using sensor information detected by a sensor device of the information processing terminal, and performs the ortho-positioning processing. An information processing system characterized by the above.

2. An information processing system for image information obtained by photographing a display shelf, wherein the information processing system comprises an image information reception processing unit configured to receive input of image information in which a display shelf to be processed is imaged, and an ortho-positioning processing unit configured to perform ortho-positioning processing for deforming the image information to be processed such that the display shelf in the input image information to be processed is positioned directly facing the imaging device or the information processing terminal that has photographed the display shelf, and the ortho-positioning processing unit performs the ortho-positioning processing using distance information to the imaging object obtained from a sensor device configured to measure the distance to the imaging object. An information processing system characterized by the above.

3. The ortho-positioning processing unit converts the distance information to the imaging object obtained from a sensor device configured to measure the distance to the imaging object into three-dimensional information, and performs the ortho-positioning processing. The information processing system according to claim 2, characterized by the above.

4. The ortho-positioning processing unit receives input of information on the pitch angle and roll angle of the imaging device that has photographed the display shelf or the information processing terminal, calculates a regression line of a point group including a plurality of points using the distance information, the pitch angle information, and the roll angle information, estimates information on the yaw angle of the imaging device or the information processing terminal, and deforms the image information using the estimated yaw angle information so as to be positioned directly facing the imaging device. The information processing system according to claim 2 or claim 3, characterized by the above.

5. The ortho-positioning processing unit Convert the distance information to an arbitrary point on the imaging object into three-dimensional information, rotate the point based on the pitch angle and roll angle information about the point, and calculate the regression line of the point cloud when the point cloud including the point is projected onto a horizontal plane, thereby estimating the yaw angle information. The information processing system according to claim 4, characterized in that.

6. The information processing system is Using a machine learning network generated using annotation data annotating a product display area and / or a price tag area in a display shelf, a recognition processing unit that recognizes a product display area and / or a price tag area in the display shelf shown in the image information subjected to the alignment process. The information processing system according to claim 2 or claim 3, characterized in that it has.

7. The information processing system is A learning processing unit that generates a machine learning network, A detection processing unit that performs detection processing of a predetermined area from image information using the network, and has, The learning processing unit is An annotation data reception processing unit that receives an input of annotation data annotating a product display area and / or a price tag area in the display shelf, A learning processing unit that performs a learning process using the received annotation data and generates a machine learning network, The information processing system according to claim 6, characterized in that it has.

8. A computer is An image information reception processing unit that receives an input of image information showing a display shelf to be processed, An information processing program that functions as an alignment processing unit that executes an alignment process for deforming the image information to be processed so that the display shelf in the image information of the processing target to which the input is received is in a position directly facing the imaging device or the information processing terminal that photographed the display shelf. And The alignment processing unit is Using the sensor information detected by the sensor device of the information processing terminal, estimating the positional relationship between the information processing terminal and the target surface in three-dimensional space, and performing the alignment process. An information processing program, characterized in that.

9. A computer is An image information reception processing unit that receives an input of image information showing a display shelf to be processed, An information processing program that functions as an ortho-positioning processing unit that performs ortho-positioning processing to transform the image information to be processed, which is the object of processing that has received the input, so that the display shelf in the image information is in a position facing the imaging device or information processing terminal that has captured the display shelf. The ortho-positioning processing unit: Performs the ortho-positioning processing using distance information to the object to be imaged obtained from a sensor device that measures the distance to the object to be imaged. An information processing program characterized by the above.

Images