Information processing system

The information processing system employs machine learning and image transformation to accurately detect and correct display shelf areas within captured images, addressing the issue of adjacent shelf inclusion and improving product identification and price tag reading accuracy.

WO2024241657A1PCT designated stage expired Publication Date: 2024-11-28MARKETVISION CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/008612
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-19
Filing Date
2024-03-06
Publication Date
2024-11-28

AI Technical Summary

Technical Problem

Existing information processing systems fail to accurately detect and process the area of a display shelf when identifying products and reading price tags, as they often include adjacent shelves and products in the image capture, leading to incorrect processing targets.

Method used

An information processing system utilizing a machine learning network to detect and correct the area of interest by receiving annotation data, performing image transformation to align the image directly with the display shelf, and using orthogonalization processing to correct for angle and distance issues, allowing for precise recognition of product display areas and price tags.

Benefits of technology

Enables accurate detection and correction of display shelf areas, improving the accuracy of product identification and price tag reading by isolating the target area from adjacent information, even with imperfect image capture conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024008612_28112024_PF_FP_ABST
    Figure JP2024008612_28112024_PF_FP_ABST
Patent Text Reader

Abstract

The purpose of the present invention is to provide an information processing system that detects a region of a display shelf to be processed during an identification / price-tag-reading process performed on a commodity displayed on the display shelf. This information processing system has a learning processing unit that generates a machine learning network, and a detection processing unit that performs a process to detect a prescribed region from image information by using the network. The learning processing unit has an annotation data reception processing unit that receives input of annotation data in which a commodity display region and / or a price tag region on a display shelf are annotated, and a learning processing unit that performs a learning process using the received annotation data and generates a machine learning network. The detection processing unit has an image information reception processing unit that receives input of image information in which the display shelf to be processed is shown, and a recognition processing unit that recognizes the commodity display region and / or the price tag region on the display shelf shown in the image information by using the generated network.
Need to check novelty before this filing date? Find Prior Art

Description

Information Processing Systems

[0001] The present disclosure relates to an information processing system that detects an area of ​​a display shelf to be processed when identifying products displayed on the display shelf and reading price tags of the products. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from Patent Application No. 2023-083471, filed May 19, 2023, the entire contents of which are incorporated herein by reference.

[0003] In convenience stores, supermarkets, and other stores, it is common for products to be displayed on shelves. Therefore, by displaying multiple products on a shelf, even if one product is purchased, another person can purchase the same product. Also, products displayed in large quantities in prominent locations attract attention and sell more than other products, creating a competitive advantage / inferiority relationship. Therefore, managing where and how many products are displayed on the shelf, as well as the prices of those products, is important in product sales strategies.

[0004] Therefore, by photographing the display shelves with a photographing device such as a camera and automatically recognizing the objects in the image information, it is possible to ascertain the display positions and numbers of products, the displayed products, and the prices written on the price tags (price cards).Technologies for this purpose are disclosed in Patent Documents 1 and 2 below.

[0005] JP-A-5-334409 JP-A-5-342230

[0006] In many stores, products are displayed on shelves in a uniform manner, so it is reasonable to identify products on shelves and read their price tags on each shelf.

[0007] When photographing products displayed on display shelves with a camera, the camera takes a picture so that the left and right edges of one display shelf fit within the angle of view, but if you try to fit the shelf exactly within the angle of view, it takes time to adjust the camera angle when photographing the display shelf. Therefore, in practice, it is common to take pictures with a margin on both sides of the angle of view.

[0008] Since the image is taken with a margin on both sides of the angle of view, the captured image information will include not only the display shelf that is the intended subject of the photograph, but also other display shelves located to the left and / or right of the intended subject of the photograph and the products displayed therein. Therefore, when attempting to identify products displayed on each display shelf and read the price tags, it is necessary to exclude from the processing the display shelves on either side of the intended subject of the photograph and the products displayed therein.

[0009] However, in the prior art, including the above-mentioned patent documents, it has not been possible to accurately exclude the adjacent shelves on either side and the products displayed there from the processing target. In some cases, a person has to manually set the processing target area from the captured image.

[0010] In view of the above, an information processing system is disclosed that detects the area of ​​a display shelf to be processed when identifying products displayed on the display shelf and reading price tags.

[0011] An information processing system according to one aspect of the present disclosure is an information processing system for image information of photographs of display shelves, the information processing system having a learning processing unit that generates a machine learning network, and a detection processing unit that performs a detection process of a predetermined area from the image information using the network, the learning processing unit having an annotation data reception processing unit that accepts input of annotation data annotated with product display areas and / or price tag areas on the display shelves, and a learning processing unit that performs a learning process using the accepted annotation data to generate a machine learning network, and the detection processing unit having an image information reception processing unit that accepts input of image information showing the display shelves to be processed, and a recognition processing unit that recognizes the product display areas and / or price tag areas on the display shelves that appear in the image information using the generated network.

[0012] By using the above, it is possible to detect the area of ​​the display shelf to be processed when identifying products displayed on the display shelf and reading the price tags.

[0013] In the information processing system of one aspect described above, the detection processing unit can be configured to have a correction processing unit that performs a correction process to correct the position of an end portion when the position of the end portion is misaligned with respect to the recognized product display area and / or price tag area.

[0014] In many cases, the product display area and / or the price tag area can be recognized, but the accuracy is not 100%. Therefore, it is preferable to provide a correction process as disclosed in the present invention.

[0015] In the information processing system of one aspect described above, the correction processing unit can be configured to detect abnormal values ​​at the edges of the detected product display area and / or price tag area, and to change the area where the abnormal value is located to the position of the edge that is normally recognized.

[0016] In one aspect of the information processing system described above, the correction processing unit can be configured to determine the positions of the ends of the detected product display area and / or price tag area, and to determine as abnormal values ​​any ends that deviate from the average or median value of those positions by more than a predetermined size or a predetermined ratio.

[0017] In one aspect of the information processing system described above, the correction processing unit can be configured to detect an object near the edge of the display shelf and set the edge of the detected product display area and / or price tag area as the position of the detected object.

[0018] The correction process can be carried out as disclosed in these publications.

[0019] In one aspect of the information processing system described above, the detection processing unit has an alignment processing unit that performs an alignment process on image information that shows the display shelf to be processed and that has received the input, to transform the image information so that it is positioned in a perpendicular position, and the recognition processing unit can be configured to use the image information after the alignment process as input image information and recognize the area to be processed using the generated network.

[0020] When photographing a store's display shelves, it is sometimes impossible to ensure a sufficient distance between the display shelves to be photographed and the camera. Furthermore, photographs are often taken at an elevation or depression angle, meaning that the photograph is not taken from a direct facing position. Processing such image information as is can lead to erroneous recognition. Therefore, it is preferable to correct the photographed image information using an orientation correction processor so that it appears as if it were photographed from a direct facing position.

[0021] In one aspect of the information processing system described above, the orientation adjustment processing unit includes a sensor information input reception processing unit that receives input of pitch angle and roll angle information of the imaging device or information processing terminal that photographed the display shelf and distance information that measures the distance to the object to be photographed, and an image transformation processing unit that performs image transformation processing on the image information, and the image transformation processing unit can be configured to use the distance information and the pitch angle and roll angle information to calculate a regression line of a point cloud that includes multiple points, estimate yaw angle information of the imaging device or information processing terminal, and transform the image information so that it is positioned directly facing the imaging device using the estimated yaw angle information.

[0022] In the information processing system of one aspect described above, the image transformation processing unit can be configured to convert distance information to an arbitrary point on the object to be photographed into three-dimensional information, rotate the point based on pitch angle and roll angle information, and calculate a regression line of a point cloud when the point cloud including the arbitrary point is projected onto a horizontal plane, thereby estimating the yaw angle information.

[0023] The configurations disclosed in these documents enable correction processing without a large processing load, making correction processing possible even on low-performance portable communication terminals such as smartphones.

[0024] An information processing system according to one aspect of the present disclosure is an information processing system for image information of a display shelf, and the information processing system has an image information reception processing unit that receives input of image information showing the display shelf to be processed, and a recognition processing unit that recognizes the product display area and / or price tag area on the display shelf that is shown in the image information using a machine learning network generated using annotation data that annotates the product display area and / or price tag area on the display shelf.

[0025] The same technical effect can be obtained by configuring as disclosed herein.

[0026] The information processing system disclosed above can be realized by loading and executing an information processing program according to one embodiment of the present disclosure into a computer. That is, the information processing program can be configured to cause a computer to function as a learning processing unit that generates a machine learning network and a detection processing unit that performs a process of detecting a predetermined area from image information using the network, wherein the learning processing unit causes the computer to function as an annotation data receiving processing unit that receives input of annotation data annotated with product display areas and / or price tag areas on a display shelf and a learning processing unit that performs a learning process using the received annotation data to generate a machine learning network, and the detection processing unit causes the computer to function as an image information receiving processing unit that receives input of image information showing a display shelf to be processed and a recognition processing unit that recognizes the product display areas and / or price tag areas on the display shelf that are shown in the image information using the generated network.

[0027] The information processing system disclosed above can be realized by loading and executing an information processing program according to one embodiment of the present disclosure into a computer. That is, the information processing program can be configured to cause a computer to function as an image information reception processing unit that receives input of image information showing a display shelf to be processed, and a recognition processing unit that recognizes the product display areas and / or price tag areas on the display shelf that are shown in the image information by using a machine learning network generated using annotation data that annotates the product display areas and / or price tag areas on the display shelf.

[0028] By using the information processing system of the present disclosure, it is possible to detect the area of ​​the display shelf that is the target of processing when identifying products displayed on the display shelf and reading the price tags.

[0029] 1 is a block diagram schematically showing an example of a configuration of an information processing system of the present disclosure. FIG. 1 is a block diagram schematically showing an example of a hardware configuration of a computer used in the information processing system of the present disclosure. FIG. 2 is a flowchart showing an example of a learning process in the information processing system of the present disclosure. FIG. 3 is a flowchart showing an example of a detection process in the information processing system of the present disclosure. FIG. 4 is a diagram showing an example of a case where a pillar member is used as an object of a display shelf. FIG. 5 is a diagram showing an example of a case where a gap is used as an object of a display shelf. FIG. 6 is a diagram showing an example of image information after execution of an orthogonalization process. FIG. 7 is a diagram showing an example of a recognition result. FIG. 8 is a diagram showing an example of image information in which only a product display area is output as a recognition result. FIG. 9 is a diagram showing an example of image information in which only a price tag area is output as a recognition result. FIG. 10 is a diagram showing an example of image information after correction processing for the recognition result. FIG. 11 is a block diagram schematically showing an example of a configuration of an orthogonalization processing unit in Example 2. FIG. 12 is a flowchart showing an example of processing of an orthogonalization process in Example 2. FIG. 13 is a diagram showing an example of image information before correction captured by an imaging device. FIG. 14 is a diagram showing a schematic view of a relationship between a target surface and a three-dimensional space. FIG. 15 is an example of image information after correction. FIG. 16 is a front view of a state in which a point cloud after rotation is mapped onto a vertical plane. FIG. 10 is a top view of a state in which a point cloud after rotation is mapped onto a horizontal plane. FIG. 11 is a diagram schematically showing processing in an angle estimation processing unit. FIG. 12 is a diagram schematically showing processing in an image transformation processing unit. FIG. 13 is a diagram showing deviations in distance information when an object to be photographed has many irregularities. FIG. 14 is a diagram schematically showing random selection of a point cloud from a matrix from among point clouds on which distance information is plotted. FIG. 15 is a diagram schematically showing image information when irregularities are captured on the left and right sides of the original object to be photographed, and abnormal values ​​in distance information resulting from this. FIG. 16 is a diagram schematically showing an example of a modified example of the normalization processing in Example 2.

[0030] 1 shows a block diagram of an example of the overall processing functions of an information processing system 1 according to one embodiment of the present disclosure. The information processing system 1 functions using an information processing terminal 2. The information processing terminal 2 may be a portable communication terminal such as a smartphone or tablet computer, a laptop computer, a desktop computer, or any other type of computer, such as a server. The appearance or name of the information processing terminal 2 is not important as long as it has the functions necessary for the processing of the information processing system 1 according to one embodiment of the present disclosure.

[0031] 2 schematically shows an example of the hardware configuration of the information processing terminal 2 in the information processing system 1. The information processing terminal 2 has a calculation device 70 such as a CPU that executes calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 that allows information to be input, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0032] When the information processing terminal 2 is equipped with a touch panel display, the display device 72 may be integrally configured with the input device 73. Touch panel displays are often used in information processing terminals 2 such as tablet computers and smartphones, but are not limited to these.

[0033] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0034] The functions of the means in the present disclosure are only logically distinct, and may be physically or practically in the same area. The order of the processes in the means in the present disclosure may be changed as appropriate. In addition, some of the processes may be omitted.

[0035] The information processing system 1 does not need to execute its processing on a single information processing terminal 2, and functions may be distributed and executed on multiple information processing terminals 2. For example, the processing of the learning processing unit 20 and the processing of the detection processing unit 21, which will be described later, may be executed on different information processing terminals 2. Either or both may be executed on a cloud server.

[0036] The information processing terminal 2 includes a learning processing unit 20 and a detection processing unit 21 .

[0037] The learning processing unit 20 receives input of image information showing display shelves, and executes a process of learning a network (learning model) for detecting the area of ​​the display shelves to be processed when identifying products displayed on the display shelves and reading the price tags. That is, the learning processing unit 20 uses deep learning such as machine learning on the image information in the detection processing unit 21, which will be described later, to generate a network (learning model) for executing a process of detecting the area to be processed from the display shelves shown in the image information.

[0038] The learning processing unit 20 includes an annotation data reception processing unit 201 and a network generation processing unit 202 .

[0039] The annotation data reception processing unit 201 receives input of annotation data to be used for network training. Examples of annotation data include image information showing a display shelf in a store and image information in which elements that make up the display shelf, such as a product display area and a price tag (price card) area, are set as annotations. For the product display area, price tag area, etc. in the annotation data, image information showing the entire product display area, price tag area, etc. is used as annotation data, and it is preferable that the positions of the left and right ends of the product display area and price tag area are set as accurately as possible. Annotations in the annotation data are not limited to those described above, and any element that makes up the display shelf can be set as an annotation.

[0040] The product display area and price tag area to be set as annotations can be set in any way, such as by the coordinates of four rectangular points that make up the area, or by the coordinates of two points, the top left and bottom right or the top right and bottom left, or by the coordinates of one point and the lengths in the x and y directions from that point. Furthermore, it is preferable that the annotation area is set on image information that has been processed, such as by keystone correction, so that the image information used as annotation data is image information taken from a directly facing position.

[0041] The network generation processing unit 202 executes a deep learning learning process by a well-known deep learning method using the annotation data received as input by the annotation data reception processing unit 201, to generate a network (learning model). The network generated by the network generation processing unit 202 is recognized by a recognition processing unit 303 in the detection processing unit 21, which will be described later.

[0042] The detection processing unit 21 detects an area to be processed from the image information to be processed when identifying and reading the price tags of products displayed on a display shelf, using the network generated by the learning processing unit 20. The detection processing unit 21 has an image information reception processing unit 301, an orthogonalization processing unit 302, a recognition processing unit 303, a correction processing unit 304, and an output processing unit 305.

[0043] The image information reception processing unit 301 receives input of image information to be processed. The image information in this case may be a single image of the entire display shelf, or may be image information of the entire display shelf obtained by photographing the display shelf with multiple images and stitching them together into a single image. Furthermore, the image information may be image information of only a portion of the display shelf.

[0044] The orthogonalization processing unit 302 performs a correction process (orthogonalization process) on the image information to be processed, which has been received as input by the image information receiving processing unit 301, so that the image information is positioned in a normal orientation. The orthogonalization processing unit 302 may perform any process as long as it can correct the image information so that the image information is positioned in a normal orientation. For example, it may perform a known keystone correction process.

[0045] When photographing a display shelf in a store, it may be difficult to ensure a sufficient distance between the display shelf to be photographed and the photographing device due to circumstances such as other display shelves being located behind the photographer and narrow aisles. Furthermore, since the display shelf is located above or below the photographer's line of sight, photographing is often performed at an elevation or depression angle. As a result, photographing is often not performed from a direct viewing position. The photographed image information is corrected by the orientation correction processing unit 302 to reflect a direct viewing position, as if the photograph had been taken from a direct viewing position.

[0046] The orthogonalization processor 302 may generate image information that has been orthogonalized to the image information to be processed, or may output parameters (correction parameters) for transforming the image information so that it is positioned in a normal position. For example, the orthogonalization processor 302 may output parameters (parameters (correction parameters) used for projective transformation) for generating orthogonalized image information so that the image information is projectively transformed to a normal position.

[0047] If the image information received as input by the image information receiving processor 301 is image information captured from a frontal position, the processing by the orthogonalization processor 302 does not need to be performed.

[0048] The recognition processing unit 303 recognizes an area where product identification / price tag reading processing is performed from the image information to be processed that has been orthogonalized by the orthogonalization processing unit 302, where the product displayed on the display shelf is identified. This recognition processing can use deep learning to execute processing to recognize each of the above-mentioned areas on the image information to be processed that has been orthogonalized by the orthogonalization processing unit 302. In this case, the image information orthogonalized by the orthogonalization processing unit 302 is input to a network (learning model) in which the weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and the area where product identification / price tag reading processing is performed is recognized from the display shelf shown in the image information. A network generated by the learning processing unit 20 can be used as the network (learning model).

[0049] The correction processing unit 304 executes a process to correct the position of the left and right edges of the area recognized by the recognition processing unit 303 where product identification / price tag reading processing is performed from the display shelf if they are misaligned.

[0050] The correction processing unit 304 can use, for example, the following processing.

[0051] The first correction process is a method of detecting abnormal values ​​at the left and right ends of an area and aligning the area containing the abnormal value with the left and right end positions of other areas. In other words, the majority of areas recognized by the recognition processing unit 303 are correctly detected. Therefore, the positions of the left and right ends of the recognized area are determined, and if it is determined that there is an abnormal value (for example, the position of the left or right end that deviates from the average value or median value of the recognized area by more than a predetermined size or a predetermined ratio), the positions of the left and right ends of the area containing the abnormal value are changed to the average value, median, etc. of the other recognized areas.

[0052] The second correction process is a method of detecting objects near the edge of a display shelf. That is, various objects may be attached near the edge of a display shelf, such as a pillar of the display shelf, a flag-like display that guides products, a gap in the display shelf, or a support member that supports products. Examples of objects are shown in Figures 5 and 6. Figure 5 shows a pillar as the object, and Figure 6 shows a gap in the display shelf as the object. This method allows the learning processing unit 20 to learn these objects as annotation data, and then detects them.

[0053] Since the left and right ends of a display shelf are common to each shelf level, by detecting the objects at the left and right ends, the left and right ends of the area recognized by the recognition processing unit 303 can be set as the positions of the objects, thereby correcting the positions of the left and right ends of the area recognized by the recognition processing unit 303. For example, the positions of the left and right ends of the area recognized by the recognition processing unit 303 can be corrected by calculating the average and median of the positions of the left and right ends of the area recognized by the recognition processing unit 303 and the position of the recognized object.

[0054] The output processing unit 305 outputs the area recognized by the recognition processing unit 303 or the area after correction processing by the correction processing unit 304 as a detection result.

[0055] Next, an example of processing using the information processing system 1 of the present disclosure will be described with reference to the flowcharts of FIGS.

[0056] First, the learning process will be described with reference to the flowchart of FIG.

[0057] Image information showing a display shelf displayed in a store and image information in which elements constituting the display shelf, such as a commodity display area where commodities are displayed and a price tag area where price tags are placed, are set as annotation data are input to an information processing terminal 2.

[0058] The input annotation data is received by the annotation data receiving processing unit 201 in the learning processing unit 20 (S100). It is preferable to receive as much annotation data as necessary for learning.

[0059] Then, the network generation processing unit 202 uses the annotation data received as input by the annotation data reception processing unit 201 to generate a network (learning model) using a known deep learning learning processing method (S110).

[0060] By performing the above processing, a network (learning model) can be generated.

[0061] Next, the detection process will be described with reference to the flowchart of FIG.

[0062] Image information for performing identification processing / price tag reading processing of products displayed on the display shelves is input as image information to be processed to the information processing terminal 2. The image information to be processed is input and accepted by the image information acceptance processing unit 301 of the detection processing unit 21 (S200).

[0063] Then, the orthogonalization processing unit 302 executes the orthogonalization process on the image information to be processed that has been received as input (S210). An example of the image information after the orthogonalization process is shown in FIG.

[0064] The normalized image information to be processed is input as input image information to the recognition processing unit 303, and the network (learning model) generated by the learning processing unit 20 is used to perform deep learning recognition processing of the elements that make up the display shelf, such as the product display area and price tag area (S220). An example of this recognition result is shown in Figure 8. In Figure 8, the product display area and price tag area are displayed in a single piece of image information, but the product display area and price tag area may be output as separate pieces of image information. Figure 9 shows the product display area alone, and Figure 10 shows the price tag area alone.

[0065] Then, the correction processing unit 304 performs a first correction process, a second correction process, etc. on the recognition result from the recognition processing unit 303 (S230). For example, in the recognition result from S220 (FIGS. 8 and 9), the left edge of the fifth area from the top of the product display area ("5Faces") is misaligned, so the position of the left edge is corrected using the first correction process, the second correction process, etc. The results after the correction process are shown in FIGS. 11 and 12.

[0066] The output processing unit 305 outputs the result after the correction process as described above as the detection result. That is, the results shown in Fig. 11 or Fig. 12 and Fig. 10 are output as the detection result. Note that the output processing unit 305 may output the position coordinates (coordinate information, or coordinate information and length) of the product display area and price tag area in addition to outputting the image information.

[0067] As described above, the image information of the display shelves output by the output processing unit 305 is subjected to a process of identifying products and a process of reading price tags and their prices using a known method for the product display area and price tag area.

[0068] In the above-described first embodiment, the case where various known orthogonalization processing techniques capable of executing processing to achieve a correct position can be used as the orthogonalization processing in the orthogonalization processing unit 302 has been described, but a case where orthogonalization processing with a reduced processing load can be realized will be described. As a result, even in an information processing terminal 2 with low processing power such as a smartphone, the orthogonalization processing can be executed with a reduced load.

[0069] An example of the configuration of the normalization processing unit 302 in this embodiment is shown in Fig. 13, and an example of the normalization process in the normalization processing unit 302 is shown in the flowchart of Fig. 14. When performing the normalization process of the present disclosure, the information processing terminal 2 is, for example, a portable communication terminal such as a smartphone, and has an imaging device such as a camera (not shown) and various sensor devices (not shown).

[0070] The image capturing device may be any device that captures image information using visible light or the like.

[0071] The sensor device may include a sensor that detects the angle of view captured by the imaging device, a sensor that detects the pitch angle and roll angle of the information processing terminal 2 or the imaging device, and a distance measuring sensor that measures the distance from the imaging device to the object being photographed, and it is desirable to have some or all of these.

[0072] The orthogonalization processing unit 302 in this embodiment performs the orthogonalization processing by regarding the front surface of the object to be photographed as a vertical rectangle (including a square) in three-dimensional space (this surface is called the object surface), and estimating the positional relationship between the information processing terminal 2 and the object surface in three-dimensional space using sensor information detected by the sensor device of the information processing terminal 2. Fig. 15 shows an example of image information (image information before correction) photographed by the photographing device.

[0073] The orthogonalization processing unit 302 includes a sensor information input reception processing unit 3021 , an angle estimation processing unit 3022 , an image transformation processing unit 3023 , and an orthogonalized image output processing unit 3024 .

[0074] The sensor information input reception processing unit 3021 receives input of information from a sensor device, such as a sensor that detects the pitch angle and roll angle of the information processing terminal 2 or the image capture device. It also receives input of distance information between the image capture device and the image capture target from a sensor (ranging sensor) that measures the distance to the image capture target. The distance information acquired is distance information from the image capture device to each point on the image capture target.

[0075] The angle estimation processing unit 3022 converts the distance information to each point on the object to be photographed, which is received as input by the sensor information input reception processing unit 3021, into three-dimensional information, and rotates this point cloud based on the roll angle and pitch angle. Because the object surface is vertical, the result of the rotation is a plane that is roughly perpendicular to the xz plane, as shown in Figure 16. Then, by calculating a regression line on the xz plane of these point clouds, the perpendicular line drawn from the imaging device to this line is estimated as the yaw angle.

[0076] The image transformation processing unit 3023 uses the yaw angle estimated by the angle estimation processing unit 3022 to transform the image information captured by the camera so that it is positioned directly facing the camera. The shape of the target rectangle is determined by assuming four vectors with the camera as the endpoints, which represent the angle of view of the camera in three-dimensional space. The vectors are rotated based on the roll angle and pitch angle input received by the sensor information input reception processing unit 3021, and then the angle estimation processing unit 3022 calculates four intersections with a vertical plane tilted to the left or right by the yaw angle estimated. Then, the image information captured by the camera is transformed by performing a projective transformation process such as OpenCV using the calculated four intersections, thereby enabling orthogonalization of the image information from a directly facing position. Figure 17 shows an example of image information after orthogonalization of the image information of Figure 15 .

[0077] The orthogonalized image output processing unit 3024 outputs the image information transformed by the image transformation processing unit 3023. For example, it outputs the corrected image information as shown in Fig. 17. The orthogonalized image output processing unit 3024 may also output parameters for performing orthogonalization, i.e., parameters for performing projective transformation processing.

[0078] Next, the orthogonalization process of the orthogonalization processor 302 in this embodiment 2 will be described with reference to the flowchart of Fig. 14. In addition, the case where the information processing terminal 2 is a smartphone will be described.

[0079] First, a photographer photographs the store's display shelves (the object to be photographed) using the camera on a smartphone that the photographer carries. Image information captured by the smartphone camera is stored (S300). An example of this image information is shown in FIG. 15. The image information in FIG. 15 was captured from a diagonally downward angle of the display shelves, looking upward (at an elevation angle), and the shelves are slightly rotated to the left (the pitch angle is 18.8 degrees upward, and the roll angle is 4.6 degrees counterclockwise).

[0080] In addition, the sensor information input reception processing unit 3021 receives input of information measured by the sensor device, such as the pitch angle and roll angle of the smartphone or camera when capturing image information with the imaging device, and distance information to each point on the object being photographed (S300).

[0081] Distance information may be acquired from any number of locations, for example, from several tens to a hundred locations, for the object being photographed. Note that the number of locations from which distance information is acquired may be any number as long as sufficient accuracy can be obtained.

[0082] The angle estimation processing unit 3022 plots the distance information received as input by the sensor information input reception processing unit 3021 in three-dimensional space to convert the distance information into a point cloud in the three-dimensional space (S310).The angle estimation processing unit 3022 then rotates this point cloud using the pitch angle and roll angle information received as input by the sensor information input reception processing unit 3021 (S320).

[0083] To generate orthogonalized image information (keystone-corrected image information) in which the products displayed on the display shelves on the target surface are upright and the shelves are horizontal, i.e., to orthogonally orient the target surface of the image information captured by the camera so that it faces the camera head-on, it is necessary to calculate the tilt of the target surface from the orientation facing the camera head-on. Therefore, the point cloud after rotation in S320 is mapped onto a horizontal plane (x-z plane), and a regression line is calculated using a robust estimation algorithm such as RANSAC (S330). Figure 18 is a front view (x-y plane in Figure 16) of the point cloud after rotation in S320 mapped onto a vertical plane, and Figure 19 is a top view (x-z plane in Figure 16) of the point cloud after rotation in S320 mapped onto a horizontal plane. Figure 20 also shows a schematic diagram of the processing performed by the angle estimation processing unit 3022.

[0084] Then, the angle (yaw angle) of the target surface tilted from the optical axis of the camera can be calculated from this regression line. Specifically, if the slope of the regression line of the point cloud is a, then the slope a of the regression line is converted into a yaw angle by calculating Equation 1 (S340). That is, the arctangent is calculated using the slope a as an argument. (Equation 1) Yaw angle = a tan(a)

[0085] Since the angle estimation processing unit 3022 can estimate the yaw angle through the above processing, the image transformation processing unit 3023 uses the estimated yaw angle to transform the image information captured by the image capture device so that the image is positioned directly facing the image capture device. This processing is schematically shown in Figure 21.

[0086] Since the angle estimation processing unit 3022 has estimated the yaw angle and the positional relationship between the camera and the target surface in three-dimensional space has been determined, the image transformation processing unit 3023 sets four lines (vectors) in three-dimensional space connecting the four vertices formed by the horizontal plane, the camera position, and the camera's angle of view (S350).Then, these four lines (vectors) are rotated using the roll angle and pitch angle accepted as input by the sensor information input acceptance processing unit 3021 and the yaw angle estimated by the angle estimation processing unit 3022 (S360).

[0087] The image transformation processing unit 3023 calculates a quadrangle of a cross section obtained by cutting the four straight lines rotated in S360 with the target plane as a target quadrangle (S370). The image transformation processing unit 3023 performs projective transformation on the image information captured by the imaging device with respect to this target quadrangle to generate image information after orthogonalization (S380). The image information obtained by orthogonalizing the image information in FIG. 15 in the orthogonalization processing unit 302 is the image information in FIG. 17.

[0088] By executing the above processing, the processing of the orthogonalization processing unit 302 can be executed.

[0089] Then, the orthogonalized image output processing unit 3024 outputs the image information transformed by the image transformation processing unit 3023. The orthogonalized image output processing unit 3024 may output parameters for projective transformation of the target rectangle as parameters to be used during orthogonalization. By outputting the parameters, the smartphone can transmit the image information before orthogonalization and the parameters instead of the image information after orthogonalization, and a predetermined server can reproduce the orthogonalization process using the image information before orthogonalization and the parameters, thereby reducing the amount of data to be uploaded.

[0090] In addition to the above-described processing by the orthogonalization processing unit 302, the following processing can also be performed.

[0091] That is, when the object being photographed is uneven, such as a display shelf displaying products, the horizontal correlation of the unevenness of the object being photographed is much stronger than the vertical correlation. Therefore, if the left-right direction of the camera is slightly misaligned with the horizontal direction of the display shelf, the distance information for horizontally arranged objects is likely to exhibit a "step-like difference," where the objects appear close up until the middle and far away from the middle. If several such rows are mixed into the acquired distance information, the estimated result will be tilted relative to the actual object surface. This is schematically shown in Figure 22. Figure 22(a) shows image information captured of an uneven display shelf, such as a display shelf displaying products, as the object being photographed. Figure 22(a) shows that the position (point) from which distance information is acquired may be the position of a product located in the foreground or the position of a shelf with no products in the background. In this case, the point cloud positional relationship shown in Figure 22(b) is obtained, which is likely to result in tilted estimated results.

[0092] Therefore, in the orthogonalization process of this second embodiment, the target surface may be further divided into matrices of a predetermined size, and distance information may be acquired from points scattered pseudo-randomly in each matrix. For example, observation points (at one or more angles, either up, down, left, or right) may be set at random positions from the intersections of each matrix, and the distance to each observation point may be measured along these points. The measured distance information may then be converted into a point cloud in three-dimensional space along the set angles, thereby preventing bias in the measured distance information due to the regularity of the shape of the object being photographed and improving the accuracy of the estimation results. This allows for random selection of points from the matrix from among the point clouds plotted with distance information when converting the distance information into a point cloud in three-dimensional space in S310. This is schematically shown in FIG. 23.

[0093] The above processing can improve the accuracy of the estimation result.

[0094] Furthermore, the orthogonalization processor 302 may perform the following processing.

[0095] That is, when the object to be photographed is a display shelf, the photographer typically stands near the center of the display shelf and takes a photograph so that the entire left and right sides of the display shelf are captured. In this case, there is a possibility that parts of adjacent display shelves will be captured near the left and right edges of the captured image information. Figure 24 shows an example of image information illustrating this. In the case of such image information, if the left and right display shelves protrude forward or are recessed backward compared to the display shelf that was originally being photographed, the regression line in S330 may be tilted. In the case of the captured image information shown in Figure 24(a), the display shelf that is captured to the right of the display shelf that was originally being photographed protrudes forward. Therefore, when the distance information is plotted in three-dimensional space, an abnormal value appears on the right side, as shown in the top view (x-z plane) of Figure 24(b). If these point clouds are processed as they are, it will affect the slope of the regression line.

[0096] Therefore, in the orthogonalization process in the orthogonalization processor 302 of the second embodiment, when a regression line is calculated using a process such as RANSAC, the following process may be executed.

[0097] First, using a process such as RANSAC, a predetermined number of points are randomly selected from the rotated point cloud in step S320 to determine a regression line. Then, for all point clouds, the number of points whose error from the regression line falls within a threshold is used as an evaluation function, and this operation is repeated multiple times. The regression line with the largest number of points is selected as the final regression line. For example, when determining a regression line from a point cloud of 100 points, sufficient accuracy can be achieved by selecting three points as the predetermined number of points and repeating this process 500 times.

[0098] Furthermore, this evaluation function can be modified to lower the weight when selecting points on the outer edge, thereby reducing the influence of irregularities in the display shelves that are reflected near the left and right edges. For example, the weighting can be modified by counting points on the outer edge as 0.5 instead of 1.

[0099] More specifically, as shown in FIG. 25, the point cloud acquisition section is divided into a predetermined number of sections, for example, four, in the x-axis direction, and the weighting of the number of points in the outer sections is changed depending on the position on the outer edge.

[0100] The orthogonalization process in the orthogonalization processor 302 in the second embodiment can be applied not only to image information captured at an elevation angle or depression angle, but also to image information captured from the left and right directions.

[0101] In the first and second embodiments, the order of the processes can be changed as desired. Furthermore, processes can be added or deleted without changing the spirit of the present disclosure. Furthermore, since the information processing system 1 of the present disclosure in the second embodiment can execute the normalization process with a small processing load, it can be applied to a smartphone or tablet computer with low processing power as the information processing terminal 2. However, it goes without saying that the information processing terminal 2 can also be executed by a computer or server with high processing power.

[0102] By using the information processing system 1 of the present disclosure, it is possible to detect the area of ​​the display shelf that is the processing target when identifying products displayed on the display shelf and reading the price tags.

[0103] 1: Information processing system 2: Information processing terminal 20: Learning processing unit 21: Detection processing unit 70: Arithmetic unit 71: Storage device 72: Display device 73: Input device 74: Communication device 201: Annotation data reception processing unit 202: Network generation processing unit 301: Image information reception processing unit 302: Orthogonalization processing unit 303: Recognition processing unit 304: Correction processing unit 305: Output processing unit 3021: Sensor information input reception processing unit 3022: Angle estimation processing unit 3023: Image transformation processing unit 3024: Orthogonal image output processing unit

Claims

1. An information processing system for image information photographing a display shelf, the information processing system having a learning processing unit that generates a machine learning network, and a detection processing unit that performs a process of detecting a predetermined area from the image information using the network, the learning processing unit having an annotation data reception processing unit that accepts input of annotation data annotated with product display areas and / or price tag areas on the display shelf, and a learning processing unit that performs a learning process using the accepted annotation data to generate a machine learning network, the detection processing unit having an image information reception processing unit that accepts input of image information showing the display shelf to be processed, and a recognition processing unit that recognizes the product display areas and / or price tag areas on the display shelf that are shown in the image information using the generated network.

2. The information processing system of claim 1, wherein the detection processing unit has a correction processing unit that executes a correction process to correct the position of an end portion of the recognized product display area and / or price tag area when the end portion is misaligned with respect to the recognized product display area and / or price tag area.

3. The information processing system according to claim 2, wherein the correction processing unit detects abnormal values ​​at the edges of the detected product display area and / or price tag area, and changes the area containing the abnormal values ​​to the position of an edge that is normally recognized.

4. The information processing system of claim 3, wherein the correction processing unit determines the positions of the ends of the detected product display area and / or price tag area, and determines as an abnormal value any end that deviates from the average or median value of the positions by a predetermined amount or a predetermined ratio or more.

5. The information processing system according to claim 2, wherein the correction processing unit detects an object near an edge of the display shelf, and sets the edge of the detected product display area and / or price tag area as the position of the detected object.

6. The information processing system of claim 1, wherein the detection processing unit has an orientation processing unit that performs an orientation process on image information showing the display shelf to be processed that has received the input, to transform the image information so that it is in a perpendicular position, and the recognition processing unit uses the image information after the orientation process as input image information and recognizes the area to be processed using the generated network.

7. The information processing system of claim 6, wherein the orientation processing unit has: a sensor information input reception processing unit that receives input of pitch angle and roll angle information of the imaging device or information processing terminal that photographed the display shelf and distance information measuring the distance to an object to be photographed; and an image transformation processing unit that performs image transformation processing on the image information, wherein the image transformation processing unit uses the distance information and the pitch angle and roll angle information to calculate a regression line of a point cloud including a plurality of the points, thereby estimating yaw angle information of the imaging device or information processing terminal, and uses the estimated yaw angle information to transform the image information so that it is positioned directly facing the imaging device.

8. The information processing system of claim 7, wherein the image transformation processing unit converts distance information to an arbitrary point on the object to be photographed into three-dimensional information, rotates the point based on pitch angle and roll angle information, and estimates the yaw angle information by calculating a regression line of a point cloud including the point when the point cloud is projected onto a horizontal plane.

9. An information processing system for image information photographing a display shelf, the information processing system having: an image information reception processing unit that receives input of image information showing the display shelf to be processed; and a recognition processing unit that recognizes the product display areas and / or price tag areas on the display shelf shown in the image information using a machine learning network generated using annotation data annotated with the product display areas and / or price tag areas on the display shelf.

10. An information processing program characterized by causing a computer to function as: a learning processing unit that generates a machine learning network; and a detection processing unit that performs processing to detect a predetermined area from image information using the network, wherein the learning processing unit causes the computer to function as: an annotation data reception processing unit that accepts input of annotation data annotated with product display areas and / or price tag areas on a display shelf; a learning processing unit that performs learning processing using the accepted annotation data and generates a machine learning network, and the detection processing unit causes the computer to function as: an image information reception processing unit that accepts input of image information showing a display shelf to be processed; and a recognition processing unit that recognizes the product display areas and / or price tag areas on the display shelf that are shown in the image information using the generated network.

11. An information processing program that causes a computer to function as: an image information receiving processing unit that receives input of image information showing a display shelf to be processed; and a recognition processing unit that recognizes the product display area and / or price tag area on the display shelf shown in the image information using a machine learning network generated using annotation data in which the product display area and / or price tag area on the display shelf is annotated.

Citation Information

Patent Citations

  • Method for inputting merchandise image in merchandise control system

    JP1993334409A

  • Merchandize display data file preparing device

    JP1993342230A

  • Merchandise identification system

    JP2015210651A

  • Display state determination system

    JP2019185684A

  • Learning model production system, program, and method for manufacturing terminal device

    JP2020119283A