Information processing system

The information processing system corrects image information using sensor data and machine learning to accurately detect and process display shelf areas for product identification and price tag reading, addressing the challenge of adjacent shelf interference.

JP2024167099AActive Publication Date: 2024-11-29MARKETVISION CO LTD
View PDF 8 Cites 0 Cited by

Patent Information

Application Number
JP2024093812
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2024-06-10
Publication Date
2024-11-29
Estimated Expiration
2043-05-19

AI Technical Summary

Technical Problem

Existing systems struggle to accurately identify and process the area of a display shelf for product identification and price tag reading due to image information capturing adjacent shelves and products, requiring manual intervention to exclude irrelevant areas.

Method used

An information processing system that uses sensor information to estimate the positional relationship between the terminal and the target surface, performing uprightization and normalization processing to correct image information, allowing for automated detection of the display shelf area using machine learning and deep learning techniques.

Benefits of technology

Enables accurate and efficient detection of the display shelf area for product identification and price tag reading, even with imperfect camera angles, reducing processing load and enabling use on devices like smartphones.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167099000001_ABST
    Figure 2024167099000001_ABST
Patent Text Reader

Abstract

To provide an information processing system that detects a region of a display shelf to be processed during an identification / price-tag-reading process performed on a commodity displayed on the display shelf.SOLUTION: An information processing system for image information for imaging a display shelf includes: an image information reception processing unit that receives input of image information in which a display shelf to be processed appears; and a directly facing processing unit which executes directly facing processing to modify the input image information in which the display shelf to be processed appears, so as to be located in a directly facing position. The directly facing processing unit estimates a positional relationship between a target surface and an information processing terminal on a three-dimensional space, using sensor information detected by a sensor device of the information processing terminal that captured the image of the display shelf, to perform the directly facing processing.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an information processing system that detects the area of ​​a display shelf to be processed when identifying products displayed on the display shelf and reading price tags of the products. [Background technology]

[0002] In various stores such as convenience stores and supermarkets, it is common for products to be sold by placing them on display shelves. Therefore, by displaying multiple products on a display shelf, even if one product is purchased, another person can purchase the same product. Also, products that are displayed in large quantities in prominent locations attract attention and sell more than other products, resulting in a competitive advantage / inferiority relationship. Therefore, it is important in product sales strategies to manage where and how many products are displayed on the display shelves, as well as the prices of those products.

[0003] Therefore, by photographing the display shelves with a photographing device such as a camera and automatically recognizing the objects in the image information, it is possible to ascertain the display positions and numbers of products, the displayed products, and the prices written on the price tags (price cards). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 5-334409 [Patent Document 2] Japanese Patent Application Publication No. 5-342230 Summary of the Invention [Problem to be solved by the invention]

[0005] In many stores, products are displayed on shelves according to product type, so it is reasonable to perform identification processing of products displayed on shelves and reading processing of price tags on the shelves.

[0006] When photographing products displayed on display shelves with a camera, the camera takes a picture so that the left and right edges of one display shelf are included within the angle of view, but if you try to fit the shelf exactly within the angle of view, it takes time to adjust the camera angle when photographing the display shelf. Therefore, in practice, it is common to take pictures with a margin on both sides of the angle of view.

[0007] Since the image is shot with a margin on both sides of the field of view, the captured image information captures not only the display shelf that is the subject of the shot, but also other display shelves to the left and / or right of the target display shelf and the products displayed therein. Therefore, when attempting to identify products displayed on a shelf-by-shelf basis and to read price tags, it is necessary to exclude the display shelves on both sides of the target display shelf and the products displayed therein from the processing targets.

[0008] However, in the prior art, including the above-mentioned patent documents, it has not been possible to accurately exclude the adjacent shelves on the left and right and the products displayed there from the processing target. In some cases, a human being has to manually set the area to be processed from the captured image. [Means for solving the problem]

[0009] In view of the above problems, the present inventor has invented an information processing system that detects the area of ​​a display shelf that is the subject of processing when identifying products displayed on the display shelf and reading the price tags of the products.

[0010] A first invention is an information processing system for image information photographing a display shelf, the information processing system having an image information reception processing unit that receives input of image information depicting the display shelf to be processed, and an orientation processing unit that performs an orientation processing on the image information depicting the display shelf to be processed that has been received as input, to transform the image information so that it is in an opposed position, the orientation processing unit using sensor information detected by a sensor device of the information processing terminal that photographed the display shelf to estimate a positional relationship in three-dimensional space between the information processing terminal and a target surface, and performing the orientation processing.

[0011] A second invention is an information processing system for image information photographing a display shelf, the information processing system having an image information receiving processing unit that receives input of image information depicting the display shelf to be processed, and an orientation processing unit that performs an orientation process on the image information depicting the display shelf to be processed that has received the input, to transform the image information so that it is in an upright position, and the orientation processing unit performs the orientation process using distance information to the object to be photographed, obtained from a sensor device that measures the distance to the object to be photographed.

[0012] When photographing a display shelf in a store, it is sometimes impossible to ensure a certain distance between the display shelf to be photographed and the photographing device. Also, photographing is often done at an elevation or depression angle, and photographing is often not done from a position facing the display shelf directly. If such image information is processed as it is, it will lead to erroneous recognition. Therefore, it is preferable to correct the photographed image information by a normalization processing unit so that it is in a position facing the display shelf directly, as if it were photographed from a position facing the display shelf directly.

[0013] The configurations of these inventions enable correction processing that does not impose a large processing load, making correction processing possible even on portable communication terminals with low capabilities, such as smartphones.

[0014] In the above-mentioned invention, the alignment processing unit can be configured as an information processing system that converts distance information to the object to be photographed, obtained from a sensor device that measures the distance to the object to be photographed, into three-dimensional information, and performs the alignment processing.

[0015] In the above-mentioned invention, the orientation correction processing unit can be configured as an information processing system that accepts input of pitch angle and roll angle information of the imaging device or the information processing terminal that photographed the display shelf, calculates a regression line of a point cloud including a plurality of points using the distance information and the pitch angle and roll angle information, estimates yaw angle information of the imaging device or the information processing terminal, and transforms the image information so that it is positioned directly facing the imaging device using the estimated yaw angle information.

[0016] In the above-mentioned invention, the alignment processor can be configured as an information processing system that converts distance information to an arbitrary point on the object to be photographed into three-dimensional information, rotates the arbitrary point based on pitch angle and roll angle information, and calculates a regression line of a point cloud including the arbitrary point when the point is projected onto a horizontal plane, thereby estimating the yaw angle information.

[0017] The configurations of these inventions enable correction processing that does not impose a large processing load, making correction processing possible even on portable communication terminals with low capabilities, such as smartphones.

[0018] In the above-mentioned invention, the information processing system can be configured as an information processing system having a recognition processing unit that uses a machine learning network generated using annotation data annotated with the product display area and / or price tag area on the display shelf to recognize the product display area and / or price tag area on the display shelf that appears in the image information that has been subjected to the alignment processing.

[0019] By using the present invention, it is possible to detect the area of ​​the display shelf that is the subject of processing when identifying products displayed on the display shelf and reading the price tags of the products.

[0020] In the above-mentioned invention, the information processing system has a learning processing unit that generates a machine learning network, and a detection processing unit that performs detection processing of a specified area from image information using the network, and the learning processing unit can be configured as an information processing system having an annotation data reception processing unit that accepts input of annotation data annotated for the product display area and / or price tag area on the display shelf, and a learning processing unit that performs learning processing using the accepted annotation data to generate a machine learning network.

[0021] The information processing system of the first invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes a computer to function as an image information reception processing unit that receives input of image information showing a display shelf to be processed, and an orientation processing unit that executes an orientation processing for transforming the image information showing the display shelf to be processed that has received the input so as to be in an orthogonal position, and the orientation processing unit estimates a positional relationship between the information processing terminal and a target surface in a three-dimensional space using sensor information detected by a sensor device of the information processing terminal that has photographed the display shelf, and performs the orientation processing.

[0022] The information processing system of the second invention can be realized by loading and executing the information processing program of the present invention into a computer. That is, the information processing program causes a computer to function as an image information reception processing unit that receives input of image information showing a display shelf to be processed, and an orthogonalization processing unit that executes an orthogonalization processing for transforming the image information showing the display shelf to be processed that has received the input so as to be in an orthogonal position, and the orthogonalization processing unit executes the orthogonalization processing using distance information to the object to be photographed obtained from a sensor device that measures the distance to the object to be photographed. Effect of the Invention

[0023] By using the information processing system of the present invention, it is possible to detect the area of ​​the display shelf that is the subject of processing when identifying products displayed on the display shelf and reading the price tags of the products. [Brief description of the drawings]

[0024] [Figure 1] 1 is a block diagram illustrating an example of a configuration of an information processing system according to the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Diagram 3] 11 is a flowchart showing an example of a learning process in the information processing system of the present invention. [Figure 4] 10 is a flowchart showing an example of a detection process in the information processing system of the present invention. [Diagram 5] FIG. 13 is a diagram showing an example in which a pillar member is used as an object of a display shelf. [Figure 6] FIG. 13 is a diagram showing an example of a break as an object on a display shelf. [Figure 7] FIG. 13 is a diagram showing an example of image information after normalization processing is performed. [Figure 8] FIG. 11 is a diagram illustrating an example of a recognition result. [Figure 9] FIG. 13 is a diagram showing an example of image information in which only a product display area is output as a recognition result. [Figure 10] FIG. 13 is a diagram showing an example of image information in which only a price tag area is output as a recognition result. [Figure 11] FIG. 13 is a diagram showing an example of image information after correction processing is performed on a recognition result. [Figure 12] FIG. 13 is a diagram showing an example of image information after correction processing is performed on a recognition result. [Figure 13] FIG. 11 is a block diagram illustrating an example of the configuration of a normalization processing unit in the second embodiment. [Figure 14] 13 is a flowchart showing an example of a normalization process in the second embodiment. [Figure 15]FIG. 13 is a diagram showing an example of image information before correction captured by a photographing device. [Figure 16] FIG. 2 is a diagram illustrating a relationship between a target surface and a three-dimensional space. [Figure 17] 11 is an example of image information after correction. [Figure 18] FIG. 13 is a front view of the rotated point cloud mapped onto a vertical surface. [Figure 19] FIG. 13 is a top view of the rotated point cloud mapped onto a horizontal plane. [Figure 20] 11 is a diagram illustrating a process in an angle estimation processing unit. FIG. [Figure 21] 11A and 11B are diagrams illustrating processes in an image transformation processing unit. [Figure 22] 11A and 11B are diagrams illustrating deviations in distance information when an object with many projections and recesses is photographed. [Figure 23] FIG. 13 is a diagram illustrating a method of randomly selecting a group of points from a matrix, among the group of points on which distance information is plotted. [Figure 24] 10A and 10B are diagrams showing image information and resulting abnormal values ​​of distance information when unevenness is captured on the left and right sides of an intended subject to be photographed; [Diagram 25] FIG. 13 is a diagram illustrating an example of a modified example of the normalization process in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] 1 shows a block diagram of an example of the overall processing functions of an information processing system 1 of the present invention. The information processing system 1 functions with an information processing terminal 2. The information processing terminal 2 may be various types of computers, such as a portable communication terminal such as a smartphone or tablet computer, a laptop computer, a desktop computer, or a server. The appearance and name of the information processing terminal 2 are not important as long as it has the functions necessary for the processing of the information processing system 1 of the present invention.

[0026] 2 shows an example of a hardware configuration of the information processing terminal 2 in the information processing system 1. The information processing terminal 2 has a calculation device 70 such as a CPU that executes arithmetic processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 that allows information to be input, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0027] When the information processing terminal 2 is equipped with a touch panel display, the display device 72 and the input device 73 may be integrally configured. Touch panel displays are often used in information processing terminals 2 such as tablet computers and smartphones, but are not limited thereto.

[0028] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display by a predetermined input device (such as a touch panel pen) or a finger.

[0029] The functions of the various means in the present invention are only logically distinct, and may be physically or practically the same area. The order of the processes in the various means of the present invention may be changed as appropriate. In addition, some of the processes may be omitted.

[0030] The information processing system 1 does not need to execute its processes on one information processing terminal 2, and functions may be distributed and executed on multiple information processing terminals 2. For example, the processes of the learning processing unit 20 and the detection processing unit 21, which will be described later, may be executed on different information processing terminals 2. Either or both of them may be executed on a cloud server.

[0031] The information processing terminal 2 includes a learning processing unit 20 and a detection processing unit 21.

[0032] The learning processing unit 20 receives input of image information showing a display shelf, and executes a process of learning a network (learning model) for detecting the area of ​​the display shelf to be processed when identifying products displayed on the display shelf and reading the price tags. That is, the learning processing unit 20 uses deep learning such as machine learning on the image information in the detection processing unit 21 described later to generate a network (learning model) for executing a process of detecting the area to be processed from the display shelf shown in the image information.

[0033] The learning processing unit 20 includes an annotation data reception processing unit 201 and a network generation processing unit 202 .

[0034] The annotation data reception processing unit 201 receives input of annotation data to be used for learning the network. Examples of annotation data include image information showing a display shelf in a store and image information in which the elements constituting the display shelf, such as a product display area and a price tag (price card) area, are set as annotations. For the product display area, price tag area, etc. in the annotation data, image information showing the entire product display area and price tag area is used as annotation data, and it is preferable that the positions of the left and right ends of the product display area and price tag area are set as accurately as possible. The annotations in the annotation data are not limited to those mentioned above, and any element constituting the display shelf can be set as an annotation.

[0035] The product display area and price tag area to be set as annotations can be set in any way, such as the coordinates of four rectangular points that make up the area, or the coordinates of two points, the top left and bottom right, or the top right and bottom left, or the coordinates of one point and the length in the x and y directions from that point, etc. In addition, it is preferable that the annotation area is set on image information that has been processed, such as keystone correction, so that the image information to be used as annotation data is image information taken from a directly facing position.

[0036] The network generation processing unit 202 executes a learning process of deep learning by a well-known learning method in deep learning using the annotation data received as input by the annotation data receiving processing unit 201, and generates a network (learning model). The network generated by the network generation processing unit 202 is recognized by a recognition processing unit 303 in the detection processing unit 21, which will be described later.

[0037] The detection processing unit 21 detects an area to be processed when identifying and reading the price tags of products displayed on a display shelf from the image information to be processed, using the network generated by the learning processing unit 20. The detection processing unit 21 has an image information reception processing unit 301, an orientation processing unit 302, a recognition processing unit 303, a correction processing unit 304, and an output processing unit 305.

[0038] The image information reception processing unit 301 receives input of image information to be processed. The image information in this case may be a single piece of image information showing the entire display shelf, or may be image information obtained by photographing the display shelf with multiple pieces of image information and stitching them together to show the entire display shelf. Furthermore, the image information may be image information showing only a part of the display shelf.

[0039] The orthogonalization processing unit 302 performs a correction process (orthogonalization process) on the image information to be processed, the input of which is received by the image information receiving processing unit 301, so as to position the image information in a right-facing position. The orthogonalization processing unit 302 may perform any process as long as it can correct the image information so as to position the image information in a right-facing position. For example, it may perform a known keystone correction process.

[0040] When photographing a display shelf in a store, it may not be possible to ensure a sufficient distance between the display shelf to be photographed and the photographing device due to circumstances such as other display shelves being located behind the photographer and narrow aisles. In addition, since the display shelf is located above or below the photographer's line of sight, photographing is often performed at an elevation or depression angle. For this reason, photographing is often not performed from a position facing the display shelf directly. The photographed image information is corrected by the orientation correction processing unit 302 so that it is positioned as if it were photographed from a position facing the display shelf directly.

[0041] The orthogonalization processor 302 may generate image information that has been subjected to orthogonalization processing for the image information to be processed, or may output parameters (correction parameters) for transforming the image information so that it is in a right-facing position. For example, the orthogonalization processor 302 may output parameters (parameters (correction parameters) used for projective transformation) for generating orthogonalized image information so that the image information is projectively transformed to be in a right-facing position.

[0042] If the image information received by the image information receiving / processing unit 301 is image information captured from a position facing the object, the processing by the orthogonalization processing unit 302 does not need to be performed.

[0043] The recognition processing unit 303 recognizes an area where the identification / price tag reading process of the commodity displayed on the display shelf is performed from the image information to be processed that has been orthogonalized by the orthogonalization processing unit 302. As the recognition process, the recognition process can be performed by using deep learning on the image information to be processed that has been orthogonalized by the orthogonalization processing unit 302 to recognize each of the above-mentioned areas. In this case, the image information orthogonalized by the orthogonalization processing unit 302 is input to a network (learning model) in which the weighting coefficient between neurons in each layer of a neural network consisting of many intermediate layers is optimized, and the area where the identification / price tag reading process of the commodity is performed from the display shelf shown in the image information is recognized. As the network (learning model), a network generated by the learning processing unit 20 can be used.

[0044] The correction processing unit 304 executes processing to correct the position of the left and right ends of the area recognized by the recognition processing unit 303 for performing the process of identifying products from the display shelf and reading the price tags, if the left and right ends are misaligned.

[0045] The correction processing unit 304 can use, for example, the following processing.

[0046] The first correction process is a method of detecting abnormal values ​​at the left and right ends of an area, and adjusting the area containing the abnormal value to the left and right end positions of other areas. That is, the majority of areas recognized by the recognition processing unit 303 are correctly detected. Therefore, the positions of the left and right ends of a recognized area are determined, and if it is determined that there is an abnormal value (for example, the position of the left and right ends that deviates from the average value or median value of the recognized area by a predetermined size or a predetermined ratio or more), the positions of the left and right ends of the area containing the abnormal value are changed to the average value, median value, etc. of the other recognized areas.

[0047] The second correction process is a method of detecting objects near the edge of a display shelf. That is, various objects may be attached near the edge of a display shelf, such as a pillar member of the display shelf, a banner-like display that guides products, a gap in the display shelf, and a support member that supports products. Examples of objects are shown in Fig. 5 and Fig. 6. Fig. 5 shows a pillar member as an object, and Fig. 6 shows a gap in the display shelf as an object. This method has the learning processing unit 20 learn these objects as annotation data, and detects them.

[0048] Since the left and right ends of a display shelf are common to each shelf level, by detecting an object at the left and right ends, the left and right ends of the area recognized by the recognition processing unit 303 can be set as the position of the object, thereby correcting the position of the left and right ends of the area recognized by the recognition processing unit 303. For example, the position of the left and right ends of the area recognized by the recognition processing unit 303 is corrected by calculating the average value and median of the positions of the left and right ends of the area recognized by the recognition processing unit 303 and the position of the recognized object.

[0049] An output processing unit 305 outputs the area recognized by the recognition processing unit 303 or the area after correction processing by the correction processing unit 304 as a detection result. EXAMPLES

[0050] Next, an example of processing using the information processing system 1 of the present invention will be described with reference to the flowcharts of FIGS.

[0051] First, the learning process will be described with reference to the flowchart of FIG.

[0052] Image information showing a display shelf displayed in a store and image information in which elements constituting the display shelf, such as a product display area where products are displayed and a price tag area where price tags are placed, are set as annotations are inputted into an information processing terminal 2 as annotation data.

[0053] The input annotation data is received by the annotation data receiving processing unit 201 in the learning processing unit 20 (S100). It is preferable to receive as much annotation data as necessary for learning.

[0054] Then, the network generation processing unit 202 uses the annotation data received as input by the annotation data reception processing unit 201 to generate a network (learning model) using a known deep learning learning processing method (S110).

[0055] By performing the above-mentioned processing, a network (learning model) can be generated.

[0056] Next, the detection process will be described with reference to the flowchart of FIG.

[0057] Image information for performing identification processing / price tag reading processing of products displayed on a display shelf is input as image information to be processed to the information processing terminal 2. The image information to be processed is input and accepted by the image information acceptance processing unit 301 of the detection processing unit 21 (S200).

[0058] Then, the orthonormalization processor 302 executes the orthonormalization process on the image information to be processed that has been received as input (S210). An example of the image information after the orthonormalization process is executed is shown in FIG.

[0059] The aligned image information to be processed is input to the recognition processing unit 303 as input image information, and the network (learning model) generated by the learning processing unit 20 is used to execute a deep learning recognition process for the elements that make up the display shelf, such as the product display area and price tag area (S220). An example of this recognition result is shown in FIG. 8. In FIG. 8, the product display area and price tag area are displayed in a single piece of image information, but the product display area and price tag area may be output in separate pieces of image information. FIG. 9 shows only the product display area, and FIG. 10 shows only the price tag area.

[0060] Then, the correction processing unit 304 performs a first correction process, a second correction process, etc., on the recognition result by the recognition processing unit 303 (S230). For example, in the recognition result in S220 (FIGS. 8 and 9), the position of the left end of the fifth area from the top of the product display area ("5Faces") is shifted, so the position of the left end is corrected using the first correction process, the second correction process, etc. The results after the correction process are shown in FIG. 11 and FIG. 12.

[0061] The output processing unit 305 outputs the result after the correction process as described above as the detection result. That is, the result of Fig. 11, or Fig. 12 and Fig. 10 is output as the detection result. Note that the output processing unit 305 may output the position coordinates (coordinate information, or coordinate information and length) of the product display area and the price tag area in addition to outputting as image information.

[0062] As described above, the image information showing the display shelves and output by the output processing unit 305 is subjected to a process of identifying products and a process of reading the price tags and their prices using a known method for the product display area and price tag area. EXAMPLES

[0063] In the above-mentioned first embodiment, the case where various known orthodontic processing techniques capable of executing processing to obtain an orthogonal position can be used as the orthodontic processing in the orthodontic processing unit 302 has been described, but the case where orthodontic processing with reduced processing load is realized can be described. As a result, even in the information processing terminal 2 with low processing power such as a smartphone, the orthodontic processing can be executed with reduced load.

[0064] An example of the configuration of the normalization processor 302 in this embodiment is shown in Fig. 13, and an example of the normalization process in the normalization processor 302 is shown in the flowchart of Fig. 14. When performing the normalization process of the present invention, the information processing terminal 2 is, for example, a portable communication terminal such as a smartphone, and has an image capture device such as a camera (not shown) and various sensor devices (not shown).

[0065] The image capturing device may be any device that captures image information using visible light or the like.

[0066] The sensor device may be equipped with some or all of the following: a sensor that detects the angle of view captured by the imaging device, a sensor that detects the pitch angle and roll angle of the information processing terminal 2 or the imaging device, and a distance measuring sensor that measures the distance from the imaging device to the object being photographed.

[0067] The orthogonalization processing unit 302 in this embodiment performs the orthogonalization processing by regarding the front surface of the photographed object as a vertical rectangle (including a square) in three-dimensional space (this surface is called the object surface) and estimating the positional relationship between the information processing terminal 2 and the object surface in three-dimensional space using sensor information detected by the sensor device of the information processing terminal 2. Fig. 15 shows an example of image information (image information before correction) photographed by the photographing device.

[0068] The orthogonal alignment processor 302 includes a sensor information input reception processor 3021 , an angle estimation processor 3022 , an image transformation processor 3023 , and an orthogonal aligned image output processor 3024 .

[0069] The sensor information input reception processing unit 3021 receives input of information from a sensor that detects the pitch angle and roll angle of the information processing terminal 2 or the image capture device from the sensor device. It also receives input of distance information between the image capture device and the object to be captured from a sensor (distance measurement sensor) that measures the distance to the object to be captured. As the distance information, distance information from the image capture device to each point on the object to be captured is acquired.

[0070] The angle estimation processing unit 3022 converts the distance information to each point of the object to be photographed, which is received by the sensor information input reception processing unit 3021, into 3D information, and rotates this point group based on the roll angle and pitch angle. Since the object surface is vertical, the result of the rotation is a plane that is approximately perpendicular to the xz plane, as shown in Fig. 16. Then, by calculating the regression line of these points on the xz plane, the perpendicular line drawn from the photographing device to this line is estimated as the yaw angle.

[0071] The image transformation processing unit 3023 transforms the image information captured by the image capture device using the yaw angle estimated by the angle estimation processing unit 3022 so that the image information is positioned directly facing the image capture device. The shape of the target rectangular surface is determined by assuming four vectors with the image capture device as end points, which represent the angle of view of the image capture device in three-dimensional space, and then rotating the vectors based on the roll angle and pitch angle inputted by the sensor information input reception processing unit 3021, and then calculating four intersections with a vertical plane tilted to the left and right by the yaw angle estimated by the angle estimation processing unit 3022. Then, the image information captured by the image capture device is subjected to a projective transformation process such as OpenCV using the calculated four intersections, thereby performing an image transformation process and an orthogonalization process on the image information from the directly facing position. FIG. 17 shows an example of image information subjected to an orthogonalization process on the image information of FIG. 15.

[0072] The orthogonalized image output processing unit 3024 outputs image information transformed by the image transformation processing unit 3023. For example, it outputs image information corrected as shown in Fig. 17. The orthogonalized image output processing unit 3024 may also output parameters for performing orthogonalization, i.e., parameters for performing projective transformation processing.

[0073] Next, the orthogonal alignment process of the orthogonal alignment processor 302 in this embodiment 2 will be described with reference to the flowchart of Fig. 14. Also, a case where the information processing terminal 2 is a smartphone will be described.

[0074] First, a person taking pictures of the store's display shelves takes a picture of the store's display shelves (the object to be photographed) using the camera of a smartphone that he or she carries. The image information taken with the smartphone camera is stored (S300). An example of this image information is shown in FIG. 15. The image information in FIG. 15 was taken from a diagonally downward direction of the display shelves looking upward (elevation angle), and the shelves are slightly rotated to the left (pitch angle is 18.8 degrees upward, and roll angle is 4.6 degrees counterclockwise).

[0075] In addition, the sensor information input reception processing unit 3021 receives input of information on the pitch angle and roll angle of the smartphone or camera when the image information is captured by the imaging device, as measured by the sensor device, and distance information to each point on the object to be photographed (S300).

[0076] The distance information may be obtained from any number of points, for example, from several tens to a hundred points, for the object to be photographed. Note that the number of points from which the distance information is obtained may be any number as long as sufficient accuracy can be obtained.

[0077] The angle estimation processing unit 3022 plots the distance information received by the sensor information input reception processing unit 3021 in a three-dimensional space, and converts the distance information into a point cloud in the three-dimensional space (S310). Then, the angle estimation processing unit 3022 rotates this point cloud using the pitch angle and roll angle information received by the sensor information input reception processing unit 3021 (S320).

[0078] In order to generate orthogonalized image information (keystone correction image information) in which the products displayed on the display shelves on the target surface are upright and the shelves are horizontal, that is, in order to orthogonally orient the target surface of the image information captured by the image capture device so that it faces the camera in a straight position, it is necessary to calculate how much the target surface is tilted from the direction facing the camera. For this reason, the point cloud after the rotation in S320 is mapped onto a horizontal plane (xz plane), and a regression line is calculated using a robust estimation algorithm such as RANSAC (S330). FIG. 18 is a front view (xy plane in FIG. 16) of the point cloud after the rotation in S320 mapped onto a vertical plane, and FIG. 19 is a top view (xz plane in FIG. 16) of the point cloud after the rotation in S320 mapped onto a horizontal plane. FIG. 20 also shows a schematic diagram of the processing in the angle estimation processing unit 3022.

[0079] Then, the angle (yaw angle) of the target surface from the optical axis of the camera can be calculated from this regression line. Specifically, if the slope of the regression line of the point cloud is a, the slope a of the regression line is converted to a yaw angle by calculating Equation 1 (S340). In other words, the arctangent is calculated using the slope a as an argument. (Number 1) Yaw angle = atan(a)

[0080] Since the angle estimation processing unit 3022 can estimate the yaw angle by the above-mentioned processing, the image transformation processing unit 3023 uses the estimated yaw angle to transform the image information captured by the image capture device so that the image is positioned directly facing the image capture device. This processing is shown in FIG.

[0081] Since the angle estimation processing unit 3022 has estimated the yaw angle and the positional relationship between the camera and the target surface in three-dimensional space is now clear, the image transformation processing unit 3023 sets four lines (vectors) in three-dimensional space connecting the four vertices formed by the horizontal plane, the camera position, and the camera's angle of view (S350).Then, the image transformation processing unit 3023 rotates these four lines (vectors) using the roll angle and pitch angle received as input by the sensor information input reception processing unit 3021 and the yaw angle estimated by the angle estimation processing unit 3022 (S360).

[0082] The image transformation processing unit 3023 calculates a quadrangle of a cross section obtained by cutting the four straight lines rotated in S360 with the target plane as a target quadrangle (S370). The image transformation processing unit 3023 performs projective transformation of the image information captured by the imaging device onto this target quadrangle to generate image information after orthogonalization (S380). The image information obtained by performing the orthogonalization processing on the image information in FIG. 15 in the orthogonalization processing unit 302 is the image information in FIG. 17.

[0083] By executing the above-mentioned processing, the processing of the orthogonalization processor 302 can be executed.

[0084] Then, the rectified image output processing unit 3024 outputs the image information transformed by the image transformation processing unit 3023. The rectified image output processing unit 3024 may output parameters for projective transformation to the target rectangle as parameters used during rectification. By outputting the parameters, the smartphone can reproduce the rectification process using the image information before rectification and the parameters in a predetermined server by sending the image information and parameters before rectification instead of the image information after rectification, and the amount of data to be uploaded can be reduced.

[0085] In addition to the above-mentioned processing by the orthogonalization processor 302, the following processing can also be performed.

[0086] That is, when the object to be photographed has many projections and recesses, such as a display shelf on which products are displayed, the correlation between the projections and recesses of the object in the horizontal direction is much stronger than that in the vertical direction. Therefore, if the left-right direction of the photographing device is slightly misaligned with the horizontal direction of the display shelf, the distance information arranged side by side is likely to be close up to the middle and far from the middle, resulting in a "step-like difference." If several such rows are mixed into the acquired distance information, the estimation result will be tilted with respect to the original object surface. This is shown diagrammatically in FIG. 22. FIG. 22(a) shows image information obtained by photographing a display shelf with many projections and recesses, for example, a display shelf on which products are displayed, as the object to be photographed. FIG. 22(a) shows that the position (point) at which the distance information is acquired may be the position of a product located in the foreground, or the position of a shelf board in the back where there are no products. In that case, the positional relationship of the point cloud will be as shown in FIG. 22(b), and the estimation result will be easily tilted.

[0087] Therefore, in the normalization process in this embodiment 2, the target surface may be further divided into matrices of a predetermined size, and distance information of points scattered pseudo-randomly in each matrix may be acquired. For example, an observation point (one or more angles in either the up, down, left, or right directions) is set at a position shifted from the intersection of each matrix with randomness, and the distance to the observation point is measured along the observation point, and the measured distance information is converted into a point group in a three-dimensional space along the set angle, so that the regularity of the shape of the object to be photographed does not cause a bias in the measured distance information, and the accuracy of the estimation result can be improved. As a result, when the distance information is converted into a point group in a three-dimensional space in S310, it is possible to randomly select a point group from the matrix from the point group on which the distance information is plotted. This is shown in FIG. 23 as a schematic diagram.

[0088] Through the above processing, the accuracy of the estimation result can be improved.

[0089] Furthermore, the orthogonalization processor 302 may perform the following process.

[0090] That is, when the object to be photographed is a display shelf, the person in charge of photography usually stands near the center of the display shelf to be photographed and photographs the display shelf so that the entire left and right sides are included. In that case, usually, a part of the adjacent display shelf may be photographed near the left and right ends of the photographed image information. An example of image information showing this is shown in FIG. 24. In the case of such image information, if the left and right display shelves protrude forward or are recessed backward from the display shelf to be photographed, the regression line in S330 may be inclined. In the case of the photographed image information shown in FIG. 24(a), the display shelf that is photographed on the right side of the display shelf to be photographed protrudes forward. Therefore, when the distance information is plotted in a three-dimensional space, an abnormal value occurs on the right side as shown in the top view (xz plane) of FIG. 24(b). If these point clouds are directly processed, it affects the inclination of the regression line.

[0091] Therefore, in the orthonormalization process in the orthonormalization processor 302 of the second embodiment, when a regression line is calculated using a process such as RANSAC, the following process may be executed.

[0092] First, a predetermined number of points are randomly selected from the point cloud after rotation in S320 using processing such as RANSAC to find a regression line. Then, for all point clouds, the number of points whose error from the regression line falls within a threshold is used as the evaluation function, and this operation is repeated multiple times, and the regression line with the most points is selected as the final regression line. For example, when finding a regression line from a point cloud of 100 points, a regression line is found by selecting three points as the predetermined number of points, and sufficient accuracy can be obtained by repeating the process 500 times.

[0093] In addition, this evaluation function can be modified to lower the weight when selecting points on the outer edge, thereby reducing the effect of unevenness of the display shelves reflected near the left and right edges. One way to change the weight is to count points on the outer edge as 0.5 instead of 1.

[0094] More specifically, as shown in FIG. 25, the point cloud acquisition section in the x-axis direction is divided into a predetermined number of parts, for example, four parts, and the weighting of the number of points in the outer sections is changed according to their location on the outer edge.

[0095] The orthogonal alignment process in the orthogonal alignment processor 302 in the second embodiment can be applied not only to image information captured at an elevation angle or depression angle, but also to image information captured from the left and right directions.

[0096] In the first and second embodiments, the order of the processes can be changed as desired. In addition, processes can be added or deleted without changing the gist of the present invention. Furthermore, since the information processing system 1 of the present invention in the second embodiment can execute the normalization process with a small processing load, the information processing terminal 2 can be applied to a smartphone or tablet computer with low processing power, but it goes without saying that the information processing terminal 2 may be executed by a computer or server with high processing power. [Industrial Applicability]

[0097] By using the information processing system 1 of the present invention, it is possible to detect the area of ​​the display shelf that is the subject of processing when identifying products displayed on the display shelf and reading the price tags of the products. [Explanation of symbols]

[0098] 1: Information processing system 2: Information processing terminal 20: Learning processing unit 21: Detection processing unit 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74: Communication equipment 201: Annotation data reception processing unit 202: Network generation processing unit 301: Image information reception processing unit 302: Orthogonalization processing unit 303: Recognition processing unit 304: Correction processing unit 305: Output processing unit 3021: Sensor information input reception processing unit 3022: Angle estimation processing unit 3023: Image transformation processing unit 3024: Orthogonal image output processing unit

Claims

1. An information processing system for image information obtained by photographing a display shelf, The information processing system includes: an image information reception processing unit that receives input of image information showing a display shelf to be processed; a normalization processing unit that performs normalization processing on image information showing the display shelf to be processed and receives the input, to transform the image information so that the display shelf is positioned in a normal position; It has The orthogonal alignment processing unit includes: using sensor information detected by a sensor device of the information processing terminal that has photographed the display shelf, estimating a positional relationship between the information processing terminal and a target surface in a three-dimensional space, and performing the normalization process; An information processing system comprising:

2. An information processing system for image information obtained by photographing a display shelf, The information processing system includes: an image information reception processing unit that receives input of image information showing a display shelf to be processed; a normalization processing unit that performs normalization processing on image information showing the display shelf to be processed and that receives the input, to transform the image information so that the display shelf is positioned in a normal position; The orthogonal alignment processing unit includes: The normalization process is performed using distance information to the object to be photographed acquired from a sensor device that measures the distance to the object to be photographed. An information processing system comprising:

3. The orthogonal alignment processing unit includes: converting distance information to the object to be photographed, acquired from a sensor device that measures the distance to the object to be photographed, into three-dimensional information, and executing the normalization process; 3. The information processing system according to claim 2.

4. The orthogonal alignment processing unit includes: receiving input of information on a pitch angle and a roll angle of an imaging device or the information processing terminal that has imaged the display shelf; calculating a regression line of a point cloud including a plurality of points using the distance information and the pitch angle and roll angle information, thereby estimating information on a yaw angle of the imaging device or the information processing terminal; Using information on the estimated yaw angle, the image information is transformed so that it faces directly toward the image capture device.

4. The information processing system according to claim 2 or 3.

5. The orthogonal alignment processing unit includes: distance information to an arbitrary point on the object to be photographed is converted into three-dimensional information, the arbitrary point is rotated based on pitch angle and roll angle information, and a regression line of a point cloud including the arbitrary point is calculated when the point cloud is projected onto a horizontal plane, thereby estimating the yaw angle information.

5. The information processing system according to claim 4.

6. The information processing system includes: a recognition processing unit that recognizes the product display area and / or the price tag area on the display shelf that appears in the image information that has been subjected to the normalization process by using a machine learning network generated using annotation data in which the product display area and / or the price tag area on the display shelf is annotated; 4. The information processing system according to claim 2, further comprising:

7. The information processing system includes: A learning processing unit that generates a machine learning network; a detection processing unit that performs a detection process of a predetermined area from image information using the network, The learning processing unit: an annotation data reception processing unit that receives an input of annotation data annotated on a product display area and / or a price tag area of ​​the display shelf; a learning processing unit that performs a learning process using the received annotation data to generate a machine learning network; 7. The information processing system according to claim 6, further comprising:

8. Computer, an image information reception processing unit that receives input of image information showing the display shelf to be processed; An information processing program that causes the display shelf to function as a normalization processing unit that performs normalization processing on image information showing the display shelf to be processed and receives the input, so that the display shelf is positioned in a normal position, The orthogonal alignment processing unit includes: using sensor information detected by a sensor device of the information processing terminal that has photographed the display shelf, estimating a positional relationship between the information processing terminal and a target surface in a three-dimensional space, and performing the normalization process; 23. An information processing program comprising:

9. Computer, an image information reception processing unit that receives input of image information showing the display shelf to be processed; An information processing program that causes the display shelf to function as a normalization processing unit that performs normalization processing on image information showing the display shelf to be processed and receives the input, so that the display shelf is positioned in a normal position, The orthogonal alignment processing unit includes: The normalization process is performed using distance information to the object to be photographed acquired from a sensor device that measures the distance to the object to be photographed.

23. An information processing program comprising:

Citation Information

Patent Citations

  • Merchandise identification system

    JP2015210651A

  • Projector device

    JP2018031803A

  • Information processing system

    JP2019219901A

  • Image processing system

    JP2020187438A

  • Image correction processing system

    JP2020201846A