Information processing system

Abstract

To provide an information processing system that supports a photographer in taking pictures of a display shelf displaying merchandise, such that the photographer can take an image without blurring or blurring that may cause problems in image recognition processing.SOLUTION: An information processing system that supports photographing of a product sales floor includes an imaging processing unit that photographs the sales floor with a photographing device of a photographing terminal, an area detection processing unit that detects a specified area from image data photographed by the imaging processing unit, and a determination processing unit that performs image quality determination processing for part or all of the detected area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention provides an information processing system that assists a photographer in taking pictures of a display shelf displaying merchandise, so that the photographer can take an image without blurring or blurring that would hinder image recognition processing. [Background technology]

[0002] In convenience stores, supermarkets, and other stores, it is common for products to be displayed on shelves. Therefore, by displaying multiple products on a shelf, even if one product is purchased, another person can purchase the same product. Also, products displayed in large quantities in prominent locations attract attention and sell more than other products, creating a competitive advantage / inferiority relationship. Therefore, managing where and how many products are displayed on the shelf, as well as the prices of those products, is important in product sales strategies.

[0003] Therefore, by photographing the display shelves with a photographing device such as a camera and automatically recognizing the objects in the image information, it is possible to ascertain the display positions and numbers of products, the displayed products, and the prices written on the price tags (price cards).Technologies for this purpose are disclosed in Patent Documents 1 and 2 below.

[0004] Furthermore, in recent years, cameras mounted on portable communication terminals such as smartphones are sometimes used as imaging devices for photographing display shelves. An example of such an imaging device is disclosed in Patent Document 3 listed below. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 5-334409 [Patent Document 2] Japanese Patent Application Publication No. 5-342230 [Patent Document 3] Japanese Patent Publication No. 2023-074321 Summary of the Invention [Problem to be solved by the invention]

[0006] The system in Patent Document 3 aims to grasp the position information of price tags attached to a display shelf by attaching a code or the like to the display shelf in advance and taking a photo of the display shelf with a smartphone or the like. However, the system in Patent Document 3 requires that a code or the like be attached to the display shelf in advance. Therefore, it cannot be applied when a code or the like is not attached to the display shelf.

[0007] Furthermore, as in Patent Documents 1 and 2, in order to perform product recognition and the like using image information obtained by photographing a display shelf, high-quality images that are free from blur or blur are required. Therefore, when photographing a display shelf, it is necessary to photograph the shelf without blur or blur. While obvious blur or blur can be seen at the time of photographing, slight blur or blur becomes apparent when product recognition is performed from the image information obtained by photographing the display shelf. If blur or blur is found during product recognition, it becomes necessary to return to the store and photograph the display shelf again.

[0008] However, taking new photographs of the display shelves is a heavy burden, and the display conditions and prices may have changed since the first photograph was taken, which is undesirable from the perspective of understanding the display conditions and prices.

[0009] Therefore, when photographing a display shelf or the like, if the image information is out of focus or blurred, which may hinder image recognition processing, it would be ideal to be able to issue a warning to the photographer at the time of shooting, but no such system has existed to date. [Means for solving the problem]

[0010] In view of the above problems, the present inventor has invented an information processing system that assists photographers in taking photographs of display shelves displaying merchandise, so that the photographs are free of out-of-focus or blurry images that would hinder image recognition processing.

[0011] The first invention is an information processing system that supports photographing a product sales floor, the information processing system having a photographing processing unit that photographs the sales floor using a photographing device of a photographing terminal, an area detection processing unit that detects a predetermined area from image data photographed by the photographing processing unit, and a judgment processing unit that performs image quality judgment processing for part or all of the detected area.

[0012] The configuration of the present invention can assist the photographer in taking pictures of display shelves on which products are displayed, so that the pictures are not out of focus or blurred.

[0013] In the above-mentioned invention, the judgment processing unit can be configured as an information processing system that sets a verification area in the detected area to be the target of the image quality judgment processing, and if the image quality of the verification area satisfies specified conditions, stores the image data as image data to be processed.

[0014] In the above-mentioned invention, the judgment processing unit can be configured as an information processing system that prompts or causes re-capture of the image data if the image quality of the verification area does not satisfy the specified condition.

[0015] As in the present invention, by determining the image quality before the analysis processing in the analysis processing unit, the image quality of the image data to be processed can be maintained at a certain level or higher. Therefore, it is possible to reduce the need to analyze image data that will be re-captured, and the burden of re-capture can be reduced.

[0016] In the above-mentioned invention, the judgment processing unit can be configured as an information processing system that, if the image quality of the verification area does not satisfy the specified conditions, further displays a message prompting the user to choose whether to store the image data as image data to be processed or to re-photograph the image, and accepts that instruction.

[0017] As in the present invention, the photographer may be allowed to select whether to use the image data for processing or to re-photograph the image. In this way, if the image quality is such that it is unclear whether it can be analyzed, the photographer can decide at their own discretion.

[0018] In the above-mentioned invention, the judgment processing unit can be configured as an information processing system that sets multiple verification areas, judges the image quality for each of the multiple verification areas, and changes the processing of the image data depending on the number of image qualities judged for each verification area.

[0019] In the above-mentioned invention, the judgment processing unit can be configured as an information processing system that judges the image quality of each of the multiple verification areas set as one or more of normal, blurred, out of focus, or slightly out of focus, and if the number of verification areas judged to have normal image quality is a majority or a large number, stores the image data as image data to be processed.

[0020] In the above-mentioned invention, the judgment processing unit can be configured as an information processing system that judges the image quality of each of the plurality of set verification areas as one or more of normal, blurred, out of focus, or slightly out of focus, and if the number of verification areas judged to have blurred or out of focus image quality is in the majority or a large number, prompts or causes the image data to be re-photographed.

[0021] In the above-mentioned invention, the judgment processing unit can be configured as an information processing system that judges the image quality of each of the multiple set verification areas as one or more of normal, blurred, out of focus, or slightly out of focus, and if the number of verification areas judged to have the image quality as slightly out of focus is in the majority or a large number, displays a message prompting the user to choose whether to store the image data as image data to be processed or to re-photograph the image, and accepts that instruction.

[0022] The image quality determination process may be configured as in these inventions.

[0023] In the above-mentioned invention, the judgment processing unit can be configured as an information processing system that sets the verification area near one or more of the top, bottom, left, right, or center edges of the area detected by the area detection processing unit.

[0024] Generally, images captured by a photographing device tend to be in focus near the center, and as the image moves away from the center, it becomes more likely to become out of focus, blurred, etc., and the effects of these become greater. Therefore, by configuring the present invention, it is possible to detect out of focus, blurred, etc. in image parts away from the center and use this to judge image quality.

[0025] In the above-mentioned invention, the information processing system has an area detection learning processing unit that generates a learning model that detects a specified area from image data captured by the shooting processing unit, and the area detection model generated by the area detection learning processing unit functions on the shooting terminal, and the image data captured by the shooting processing unit is input into the area detection model to detect the specified area, so that the information processing system can be configured as such.

[0026] In the above-mentioned invention, the information processing system has a judgment model generation processing unit that generates a judgment learning model that determines whether the image data or a portion thereof captured by the shooting processing unit satisfies image quality conditions to an analyzable degree, and the judgment model generated by the judgment learning processing unit functions on the shooting terminal, and the information processing system can be configured to judge the image quality conditions by inputting a portion or all of the image data captured by the shooting processing unit into the judgment model.

[0027] As in these inventions, by storing the learning model in the imaging device, communication delays can be eliminated, leading to faster processing.

[0028] In the above-described invention, the information processing system can be configured as an information processing system having a shooting condition processing unit that determines whether shooting conditions are met and displays a warning if it determines that the shooting conditions are not met.

[0029] According to the present invention, by displaying a warning when the conditions for photography are not met, the photographer can take a photo under appropriate conditions so that the warning does not appear.

[0030] In the above-described invention, the shooting condition processing unit can be configured as an information processing system that determines whether the object to be photographed is contained in the image data at a sufficiently large size without protruding therefrom, or the photographing orientation of the object to be photographed.

[0031] In the above-mentioned invention, the shooting condition processing unit can be configured as an information processing system that determines whether the subject to be photographed is contained in the image data at a sufficiently large size without extending beyond it, using the relationship between the position of the gap in the display shelf detected from the image data, or the width of the area detected by the area detection processing unit, and the width of the guideline in the image data.

[0032] In the above-mentioned invention, the shooting condition processing unit can be configured as an information processing system that determines whether the position of the gap in the display shelf detected from the image data, or the position of the area detected by the area detection processing unit, is located within a predetermined range of the image data, and determines whether the width direction of the area detected by the area detection processing unit is at a ratio of a certain level or greater to the width direction length of the guideline.

[0033] In the above-mentioned invention, the shooting condition processing unit can be configured as an information processing system that determines the shooting direction using a sensor provided on the shooting terminal, or determines the shooting direction by using the positional relationship of the ends of the area detected by the area detection processing unit.

[0034] Whether or not the conditions for photography are satisfied can be realized by carrying out the processing as in these inventions.

[0035] In the above-mentioned invention, the shooting condition processing unit can be configured as an information processing system that, if it determines that the shooting conditions are not met, displays a warning by changing the display of the area that does not meet the shooting conditions.

[0036] If the conditions for photography are not met, the display of the area that does not meet the conditions can be changed as in the present invention, allowing the photographer to easily recognize that area.

[0037] In the above invention, the photographing condition processing section can be configured as an information processing system in which, if it is determined that the photographing conditions are not satisfied, the image data cannot be recorded.

[0038] If the shooting conditions are not met, the image data will be wasted, so it is preferable to be able to control the image data so that it cannot be recorded in the first place.

[0039] In the above-mentioned invention, the information processing system can be configured as an information processing system having a photographed image data reception processing unit that receives image data or a portion thereof that has been designated as the data to be processed by the judgment processing unit, and an identification processing unit that identifies one or more of the product's identification information and price from the received image data or a portion thereof.

[0040] Since image data of a certain level of image quality or higher is accepted from the photographing device, the accuracy of the analysis process is improved.

[0041] A 20th invention is a photographing terminal used when photographing a product sales floor, the photographing terminal having a photographing processing unit that photographs the sales floor using a photographing device of the photographing terminal, an area detection processing unit that detects a predetermined area from image data photographed by the photographing processing unit, and a judgment processing unit that performs image quality judgment processing for part or all of the detected area.

[0042] In the above-mentioned invention, the photographing terminal has an area detection model storage unit that stores a learning model that detects a specified area from the image data, and the area detection learning processing unit can be configured like a photographing terminal that detects the specified area by inputting the image data photographed by the photographing processing unit into the area detection model and displays the area on a display device of the photographing terminal.

[0043] In the above-mentioned invention, the photographing terminal has a judgment model memory unit that stores a judgment learning model that judges whether the image data or a portion thereof satisfies predetermined image quality conditions, and the judgment processing unit can be configured like a photographing terminal that judges the image quality conditions by inputting a portion or all of the image data photographed by the photographing processing unit into the judgment model.

[0044] In the above-mentioned invention, the photographing terminal can be configured to have a photographing condition processing unit that determines whether the image data satisfies the conditions related to photographing, and if it determines that the conditions related to photographing are not satisfied, displays a warning on a display device of the photographing terminal.

[0045] The processing of the present invention is preferably performed by a photographing terminal.

[0046] The information processing system of the first invention can be realized by executing the program of the present invention on a computer. That is, the information processing program causes the computer to function as an image capture processing unit that captures an image of a product sales floor using an image capture device of an image capture terminal, an area detection processing unit that detects a predetermined area from image data captured by the image capture processing unit, and a judgment processing unit that performs image quality judgment processing on part or all of the detected area.

[0047] The photographing terminal of the twentieth invention can be realized by installing and running the program of the present invention in the photographing terminal. That is, the program causes the photographing terminal used to photograph a product sales floor to function as a photographing processing unit that photographs the sales floor with the photographing device of the photographing terminal, an area detection processing unit that detects a predetermined area from image data photographed by the photographing processing unit, and a judgment processing unit that performs image quality judgment processing on part or all of the detected area. [Effects of the Invention]

[0048] By using the information processing system of the present invention, it is possible to assist the photographer in taking pictures of display shelves displaying merchandise, so that the pictures are free of out-of-focus or blurring that would hinder image recognition processing. [Brief explanation of the drawings]

[0049] [Figure 1] 1 is a block diagram schematically illustrating an example of a configuration of an information processing system according to the present invention. [Figure 2] FIG. 2 is a block diagram schematically illustrating an example of a hardware configuration of a computer used in the information processing system of the present invention. [Figure 3] 3 is a flowchart showing an example of the overall processing in the information processing system of the present invention. [Figure 4] FIG. 10 is a diagram schematically illustrating an outline of the processing in the determination processing unit in the case of a regular sales area. [Figure 5] FIG. 10 is a diagram schematically illustrating an overview of the processing in the determination processing unit in the case of a special sales area. [Figure 6]FIG. 2 is a diagram schematically illustrating a verification region. [Figure 7] FIG. 10 is a diagram schematically illustrating an example of another process for searching a verification region. [Figure 8] FIG. 10 is a diagram showing an example of a screen when photographing a standard sales area. [Figure 9] FIG. 9 is a diagram showing an example of a screen in a state where area detection processing for a product display area and a price tag area has been performed from the screen of FIG. 8. [Figure 10] FIG. 10 is a diagram showing an example of a screen when the photographing terminal is held horizontally. [Figure 11] FIG. 10 is a diagram schematically illustrating a verification region to be input as an input value to a determination model for a regular sales area. [Figure 12] FIG. 10 is a diagram showing an example of a screen when the determination processing unit determines that the image has been captured normally. [Figure 13] FIG. 10 is a diagram showing an example of a screen displayed when a determination processing unit determines that re-imaging is to be performed. [Figure 14] FIG. 10 is a diagram showing an example of a screen displayed when the determination processing unit determines that the decision as to whether to retake a photograph is left to the photographer. [Figure 15] FIG. 10 is a diagram showing an example of a screen when photographing a special event sales area. [Figure 16] FIG. 10 is a diagram schematically illustrating a verification region to be input as an input value to a determination model for a special sales area. [Figure 17] FIG. 10 is a block diagram illustrating an example of the configuration of an information processing system according to a third embodiment. [Figure 18] 11 is a flowchart illustrating an example of the overall processing in the information processing system according to the third embodiment. [Figure 19] FIG. 10 is a diagram showing an example of a screen for determining whether the entire display shelf to be photographed is captured in a sufficiently large size without extending beyond the image data. [Figure 20] FIG. 10 is a diagram showing an example of a screen displaying a warning when "oblique shooting" is performed. [Figure 21] FIG. 10 is a diagram showing an example of a screen displayed when the imaging condition processing unit determines that all conditions are met. DETAILED DESCRIPTION OF THE INVENTION

[0050] An example of the overall processing functions of an information processing system 1 of the present invention is shown in a block diagram in Fig. 1. The information processing system 1 uses a management terminal 2 and a photography terminal 3. The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. The photography terminal 3 is a terminal operated by a photographer who takes pictures of product display locations, etc.

[0051] In the present invention, a display location refers to a place where products are displayed, and mainly includes a regular sales area and a special sales area. A regular sales area is a place where products are displayed by placing or hanging products on each shelf of standardized racks (shelves) of normal size. A special sales area is a place where products are displayed by stacking products in a pyramid shape or by placing or hanging products in cardboard boxes or baskets. Regular sales areas are often, but not limited to, arranged along the walls of a store or in a regular row in a predetermined direction within the store. They may also be installed near the cash register. A special sales area is often, but not limited to, arranged near the longitudinal end of a regular sales area where baskets, cardboard boxes, etc. are placed in the aisle and multiple racks are regularly arranged.

[0052] The management terminal 2 and the photographing terminal 3 in the information processing system 1 are realized using a computer. An example of the hardware configuration of a computer is shown in Figure 2. The computer has a calculation device 70 such as a CPU that executes the calculation processing of a program, a storage device 71 such as a RAM or a hard disk that stores information, a display device 72 such as a display that displays information, an input device 73 such as a keyboard or a mouse that can input information, and a communication device 74 that transmits and receives the processing results of the calculation device 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0053] If the computer is equipped with a touch panel display, the display device 72 may be integrated with the input device 73. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0054] The touch panel display is a device that integrates the functions of the display device 72 and the input device 73 in that input can be made directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0055] The photographing terminal 3 may be equipped with a photographing device such as a camera in addition to the above-mentioned devices. A portable communication terminal such as a mobile phone, a smartphone, or a tablet computer may also be used as the photographing terminal 3. The photographing terminal 3 photographs a predetermined subject and inputs image data of the photographed subject (photographed image data) to the management terminal 2. For example, a display shelf in a store is photographed and image data of the display shelf (photographed image data) is input to the management terminal 2. In this case, products are displayed on the display shelf, and the display shelf is photographed.

[0056] The functions of the various means in the present invention are only logically distinct, and may be physically or practically the same area. The order of the processes in the various means of the present invention may be changed as appropriate. Also, some of the processes may be omitted. For example, the normalization process described below may be omitted. In that case, the process may be performed on image data that has not been normalized.

[0057] The management terminal 2 of the information processing system 1 includes an area detection model generation processing unit 20, a determination model generation processing unit 21, and an analysis processing unit 22.

[0058] The area detection model generation processing unit 20 generates a machine learning model that performs area detection from image data captured by the photographing terminal 3. Areas targeted for machine learning here include an area where products are displayed (product display area), an area where price tags (price cards) are placed (price tag area), and an area where event shelves are placed (event shelf area 102). Product display areas include, for example, the shelf area of ​​a display shelf in a regular sales area, and a product hanging area (an area where products such as toothbrushes are hung and displayed). Price tag areas include, for example, price tags (price cards) attached to shelf shelves of a display shelf in a regular sales area, or an area where a price tag can be attached (the area in front of the shelf). Event shelf areas 102 include, for example, an area where products are displayed in a special sales area, and an area where product price tags are placed. Unlike the product display area and price tag area of ​​the regular sales floor, the event shelf area 102 is generally irregular in shape, so rather than distinguishing between the product display area where products are displayed and the price tag area where price tags are placed, it is preferable to detect the event shelf area 102 as a combination of both, but it is also possible to detect the area by distinguishing between the product display area and the price tag area.

[0059] As will be described later, when a regular sales floor or special sales floor is photographed with the photographing device of the photographing terminal 3, the photographed image data is input as an input value to a learning model (network) generated by the area detection model generation processing unit 20, thereby detecting various areas such as product display areas, price tag areas, and special event shelf areas 102 that appear in the image data. The learning model for this area detection is generated by the area detection model generation processing unit 20.

[0060] It is preferable that deep learning be used as machine learning in the region detection model generation processing unit 20. In this case, the learning model is a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized. Note that the machine learning used by the region detection model generation processing unit 20 is not limited to deep learning, and other machine learning methods can also be used.

[0061] The area detection model generation processing unit 20 includes an area detection learning data storage unit 200 and an area detection learning processing unit 201 .

[0062] The area detection learning data storage unit 200 stores learning data (area detection learning data) for machine learning in the area detection learning processing unit 201, which will be described later. The learning data used in this case includes image data of various regular sales areas and special sales areas, and annotation data in which product display areas and price tag areas are tagged in the image data.

[0063] The area detection learning processing unit 201 inputs the learning data stored in the area detection learning data storage unit 200 as correct answers to a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized, and causes the learning model to learn. A known machine learning method can be used.

[0064] It is preferable that the learning model for area detection (area detection model) generated by the area detection learning processing unit 201 is downloaded to the photography terminal 3 together with an application program for photographing the display location, as described below.

[0065] It is preferable that the area detection learning processing unit 201 generates an area detection model for a regular sales floor and an area detection model for a special sales floor as learning models for the area detection models. Therefore, the area detection learning data storage unit 200 generates an area detection model (area detection model for regular sales floor) trained by machine learning using image data of the regular sales floor and annotation data tagged with the product display area and price tag display area therein, and an area detection model (area detection model for special sales floor) trained by machine learning using image data of the special sales floor and annotation data tagged with the special shelf area 102 therein, and downloads the model to the photography terminal 3.

[0066] The judgment model generation processing unit 21 generates a machine learning model to determine whether the image data captured by the photographing terminal 3 is suitable for analysis processing by the analysis processing unit 22 described below, that is, whether all or a specified area of ​​the image data captured by the photographing terminal 3 meets image quality conditions such as whether it is out of focus or blurred to the extent that it can be analyzed by the analysis processing unit 22.

[0067] That is, as will be described later, all of the image data of the regular sales area photographed by the photographing device of the photographing terminal 3, or image data of the product display area and / or price tag area, or all of the image data of the event sales area photographed, or image data of the event shelf area 102, are input as input values ​​into a learning model (network) generated by the judgment model generation processing unit 21, thereby generating a machine learning model for determining whether the photographed image data, or the image data of the product display area and / or price tag area in the image data, or the image data of the event shelf area 102 is out of focus or blurred to an extent that it can be analyzed by the analysis processing unit 22.

[0068] It is preferable that the determination model generation processing unit 21 uses deep learning as machine learning. In this case, the learning model is a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of multiple intermediate layers are optimized. Note that the machine learning used by the determination model generation processing unit 21 is not limited to deep learning, and other machine learning methods can also be used.

[0069] The determination model generation processing unit 21 includes a determination image data storage unit 210 , a learning image data generation processing unit 211 , a determination learning data storage unit 212 , and a determination learning processing unit 213 .

[0070] The determination image data storage unit 210 stores image data for generating image data for machine learning in the learning image data generation processing unit 211, which will be described later. For example, it stores image data of various regular sales areas and special sales areas, image data in which product display areas and price tag areas have been cut out from the image data in which the regular sales areas have been taken, and image data in which the special shelf area 102 has been cut out from the image data in which the special sales areas have been taken. The image data in which the product display areas and price tag areas have been cut out from the image data in which the regular sales areas have been taken, and the image data in which the special shelf area 102 has been cut out from the image data in which the special sales areas have been taken, may actually be the product display areas, price tag areas, and special shelf areas 102 cut out from the image data in which the respective sales areas have been taken, or these areas may be identified as the areas to be processed.

[0071] It is preferable that the image data stored in the determination image data storage unit 210 is image data with image quality (sharp image quality) that is free from out-of-focus and blur to the extent that analysis can be performed by the analysis processing unit 22.

[0072] The learning image data generation processing unit 211 performs processes such as blurring and motion blurring using known methods on the image data stored in the determination image data storage unit 210. Bluring and motion blurring can be performed any number of times, but for example, image data that has been motion blurred on image data with sharp image quality, image data that has been blurred in one-pixel increments on image data with sharp image quality, and image data that has been blurred in three-pixel increments on image data with sharp image quality can be generated. Known methods can be used for motion blurring and blurring.

[0073] This results in four types of image data for the same subject: sharp image data (image data stored in the determination image data storage unit 210), image data processed with motion blur, image data processed with blurring in one-pixel increments, and image data processed with blurring in three-pixel increments. Each image data is annotated with information indicating image quality, such as Sharp (normal), Motion Blurred (blurred), Defocused (out-of-focus), or Slightly Defocused (slightly out-of-focus), and annotation data, a set of image data and annotations, is generated as training image data. The generated training image data is stored in the determination training data storage unit 212. The degree of blurring or blurring can be determined based on the degree to which the identification process (image recognition process) in the identification processing unit 222 of the analysis processing unit 22, which will be described later, is hindered, i.e., the degree to which erroneous recognition begins to occur. Defocused (out of focus (somewhat out of focus)) is out of focus to the extent that there is a high possibility of misrecognition, Slightly Defocused (slightly out of focus (somewhat out of focus)) is out of focus to the extent that there is a low possibility of misrecognition, but there is also a possibility. Motion Blurred (blurred (somewhat blurred)) is out of focus to the extent that there is a high possibility of misrecognition. And Sharp (normally sharp) means that there is almost no possibility of misrecognition. This degree can be distinguished into any number of levels and types.

[0074] The learning image data generation processing unit 211 generates image data that has been motion blurred and image data that has been blurred from image data with sharp image quality, but if there is out-of-focus or blurred image data in advance, it is sufficient to store this in the judgment learning data storage unit 212, and the judgment image data storage unit 210 and the learning image data generation processing unit 211 do not need to be provided.

[0075] Furthermore, although the learning image data generation processing unit 211 has been described above as performing blurring and motion blurring to detect out-of-focus and shaking, it may also perform other processing.

[0076] The judgment learning data storage unit 212 stores the learning image data generated by the learning image data generation processing unit 211. That is, the unit stores sharp image data of image data of various regular sales areas or image data of product display areas and price tag areas cut out from image data of regular sales areas, image data of special sales areas or image data of special shelf areas cut out from image data of special sales areas, and annotation data in which image data processed with blurring and / or motion blur are associated with image quality annotations.

[0077] The judgment learning processing unit 213 inputs the learning data (annotation data) stored in the judgment learning data storage unit 212 as correct answers to a learning model in which weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized, and causes the learning model to learn. A known machine learning method can be used.

[0078] It is preferable that the learning model for judgment (judgment model) generated by the judgment learning processing unit 213 is downloaded to the photography terminal 3 together with an application program for photographing the display location, as described below.

[0079] It is preferable that the judgment learning processing unit 213 generates a judgment model for a regular sales floor and a judgment model for a special sales floor as learning models for the judgment models. Therefore, the judgment image data storage unit 210 stores image data of the regular sales floor or image data obtained by photographing the regular sales floor, where the product display area and price tag area are cropped, and image data of the special sales floor or image data obtained by photographing the special sales floor, where the event shelf area 102 is cropped. In this case, the learning image data generation processing unit 211 generates image data by applying motion blur processing and blurring processing to the image data of the regular sales floor or image data obtained by photographing the regular sales floor, where the image data is cropped by the product display area and price tag area, and stores the image data in the judgment learning data storage unit 212 as annotation data for the regular sales floor, with an associated image quality annotation. In addition, the learning image data generation processing unit 211 generates image data by performing motion blur processing and blur processing on image data of the event sales floor or image data of the event shelf area 102 cut out from image data of the event sales floor or image data photographed of the event sales floor, and stores the image data in the judgment learning data storage unit 212 as annotation data for the event sales floor by associating it with image quality annotations.

[0080] The judgment learning data storage unit 212 stores annotation data in which sharp image data for regular sales areas, image data processed with blurring and / or motion blur, and image quality annotations are associated with each other. The judgment image data storage unit 210 stores annotation data in which sharp image data for special sales areas, image data processed with blurring and / or motion blur, and image quality annotations are associated with each other.

[0081] The determination learning processing unit 213 inputs the annotation data for the regular sales area stored in the determination learning data storage unit 212 as a correct answer and performs machine learning to generate a determination learning model for the regular sales area, and downloads it to the photographing terminal 3. The determination learning processing unit 213 also inputs the annotation data for the special sales area stored in the determination learning data storage unit 212 as a correct answer and performs machine learning to generate a determination learning model for the special sales area, and downloads it to the photographing terminal 3.

[0082] The analysis processing unit 22 receives image data (photographed image data) of the regular sales floor and the special sales floor photographed by the photographing terminal 3, and performs processing to identify the products and / or prices shown in the received photographed image data.

[0083] The analysis processing unit 22 includes a photographed image data reception processing unit 220 , a photographed image data storage unit 221 , and an identification processing unit 222 .

[0084] The photographed image data reception processing unit 220 receives photographed image data of the regular sales floor and the special sales floor photographed by the photographing terminal 3, and stores the data in the photographed image data storage unit 221. As will be described later, the received photographed image data is image data that has been determined by the determination processing unit 34 to be free from out-of-focus or blur to the extent that analysis processing by the analysis processing unit 22 is possible, and therefore has image quality that is less likely to cause errors in the analysis processing by the analysis processing unit 22.

[0085] The photographed image data storage unit 221 stores the photographed image data accepted by the photographed image data acceptance processing unit 220.

[0086] The identification processing unit 222 identifies the products in the product display area or event shelf area 102 of the photographed image data stored in the photographed image data storage unit 221, the prices of the products in the price tag area or event shelf area 102, etc. The identification processing unit 222 can use a known method. In addition to image matching, machine learning processing can be used as the process for identifying the products, prices, etc.

[0087] In the case of processing by machine learning, a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many intermediate layers are optimized is trained by inputting the products shown in the product display area, the prices on the price tags in the price tag area, and the products and price tags shown in the event shelf area 102 as correct answers. Then, by inputting the photographed image data in the photographed image data storage unit 221 into this learning model, the prices of the products shown in the product display area and / or the price tags shown in the price tag area, the prices of the products and / or the price tags shown in the event shelf area 102, etc. can be identified. The machine learning method itself can be a known method.

[0088] The photographing terminal 3 is a portable communication terminal operated by a photographer who takes photographs of the regular sales area, special sales area, etc. of a store, and photographs the products on display, their price tags, etc. Application software for realizing the processing in the photographing terminal 3 of the information processing system 1 of the present invention is installed in the portable communication terminal, and the photographer realizes the processing in the photographing terminal 3 of the present invention by starting and using the application software.

[0089] The photographing terminal 3 has a photographing processing unit 30, an area detection model storage unit 31, an area detection processing unit 32, a determination model storage unit 33, a determination processing unit , and a transmission processing unit .

[0090] The photographing processing unit 30 activates a photographing device such as a camera provided in the photographing terminal 3, and photographs the regular sales area and the special sales area with the photographing device. When the photographing range can be identified, the area detection processing unit 32 (described later) acquires image data of the regular sales area and the special sales area by releasing the shutter of the photographing device or extracting still image information from a moving image.

[0091] The area detection model storage unit 31 stores the area detection model (machine learning model) generated by the area detection model generation processing unit 20 of the management terminal 2. The area detection model storage unit 31 may store an area detection model for a regular sales area and an area detection model for a special sales area, or may store only one of them.

[0092] The area detection processing unit 32 detects product display areas and price tag areas from image data of the regular sales floor photographed by the photography processing unit 30 using a photography device, and also detects the event shelf area 102 from image data of the event sales floor photographed by the photography processing unit 30 using a photography device.

[0093] When the photography processing unit 30 receives an operation to start photography from the photographer and acquires image data photographed by the photography device, the area detection processing unit 32 inputs the image data (for example, image data of 20 frames per second, or a portion of the image data) as input values ​​into the area detection model stored in the area detection model storage unit 31, thereby being able to detect the product display area, price tag area, and event shelf area 102. It is preferable that the area detection processing unit 32 specifies the product display area, price tag area, and event shelf area 102 detected from the image data photographed by the photography processing unit 30 so that they can be cut out.

[0094] The judgment model storage unit 33 stores a judgment model (machine learning model) generated by the judgment model generation processing unit 21 of the management terminal 2. The judgment model storage unit 33 may store a judgment model for a regular sales area and a judgment model for a special sales area, or may store only one of them.

[0095] The determination processing unit 34 determines whether the image data captured by the image capture processing unit 30 is out of focus, blurred, or the like, using the determination model stored in the determination model storage unit 33. An overview of the processing in the determination processing unit 34 is shown in FIG.

[0096] The judgment processing unit 34 sets one or more verification areas 101 of a predetermined size in the image data captured by the imaging processing unit 30, and inputs the image data in the verification areas 101 as input values ​​into the judgment model in the judgment model storage unit 33, thereby judging the image quality of the verification areas 101. For example, the image quality of the image data of the verification areas 101 is judged as Sharp, Motion Blurred, Defocused, Slightly Defocused, etc.

[0097] The size of the verification area 101 can be set arbitrarily. For example, if an image is captured so that the entirety of one shelf of a display shelf fits within the angle of view of the imaging device, and the image resolution of the imaging device is approximately 3000 pixels horizontally by 4000 pixels vertically, the size of the verification area 101 can be set, for example, to 240 pixels vertically and 240 pixels horizontally when the image is captured vertically, or to 320 pixels vertically and 320 pixels horizontally when the image is captured horizontally.

[0098] The judgment processing unit 34 judges whether the image quality of the image data of each verification area 101 satisfies predetermined conditions, and displays a message indicating whether the photograph was taken successfully, whether to prompt the photographer to take a new photograph, or whether to leave the decision on whether to take a new photograph to the photographer.

[0099] For example, if the majority of the verification areas 101 in the image data are determined to have sharp image quality, it is determined that the image has been captured normally.

[0100] For example, if the majority of the verification areas 101 in the image data are judged to have Motion Blurred or Defocused image quality, the judgment processing unit 34 displays a message on the display device 72 of the photographing terminal 3 prompting the user to take a new photograph.

[0101] For example, if the majority of the verification areas 101 in the image data are judged to have slightly defocused image quality, the judgment processing unit 34 displays a message on the display device 72 of the photographing terminal 3 indicating that the photographer is free to decide whether or not to take a re-photograph.

[0102] As shown in Fig. 4, when photographing display shelves in a regular sales area, verification areas 101 of a predetermined size are set near the upper left and right edges of the product display area or price tag area on the top shelf, near the lower left and right edges of the product display area or price tag area on the bottom shelf, and near the center of the image data, and the image quality of each verification area 101 is judged. Fig. 4(a) is an example of image data of display shelves in a regular sales area photographed by the photography processing unit 30, and Fig. 4(b) is a diagram showing the verification areas 101.

[0103] 5, when photographing display shelves in a special sale section, for example, a predetermined area in a special sale section area 102 in the special sale section, which is located farthest in the left-right upper and right-left lower directions from the center of a rectangular area 103 including the special sale section area 102, is searched for and set as a verification area 101. For example, the search for the verification area 101 is performed by shifting the area by a certain distance from the center of the rectangular area 103 toward the left-right upper and right-left lower ends of the verification area 101, and when the farthest part that is in an inclusive relationship is reached, that part is set as the verification area 101. FIG. 5(a) is an example of image data of display shelves in a special sale section photographed by the photographing processing unit 30, FIG. 5(b) is a diagram showing the search process for the verification area 101, and FIG. 6 is a diagram showing the verification area 101.

[0104] Another method is to imagine 45-degree diagonal lines 104 touching the left, right, top, and bottom edges of the detection area, start searching for the verification area 101 from positions touching each of these diagonal lines 104, and gradually search the verification area 101 toward the center of a rectangular area 103 that includes the detection area. This is shown schematically in Figure 7.

[0105] If the determination processing unit 34 determines that the image was captured normally, or if an instruction that re-capture is not required is received from the photographer, the transmission processing unit 35 stores the image data captured by the image capture processing unit 30, or the image data of the product display area and / or shelf area in the image data, in a predetermined storage area.Then, the stored image data is sent to the management terminal 2 at a predetermined timing.

[0106] The image data transmitted by the transmission processing unit 35 here is image data that has been determined by the determination processing unit 34 to be free from out-of-focus or blur to the extent that analysis processing in the analysis processing unit 22 is possible, and therefore has image quality that is less likely to cause errors in the analysis processing in the analysis processing unit 22. This reduces the possibility that re-imaging is required. [Example]

[0107] Next, an example of processing using the information processing system 1 of the present invention will be described with reference to the flowchart of FIG.

[0108] First, the process of generating the region detection model will be described.

[0109] The area detection learning data storage unit 200 stores image data of various regular sales areas and annotation data in which the product display areas and price tag areas are tagged in the image data, as well as image data of various special sales areas and annotation data in which the special shelf area 102 is tagged in the image data.

[0110] The area detection learning processing unit 201 receives an operation to perform processing to generate an area detection model for a standard sales area on the management terminal 2, extracts image data of the standard sales area and annotation data tagged with product display areas and price tag display areas from that image data to be stored in the area detection learning data storage unit 200, and uses this to generate a learning model that performs area detection on the image data of the standard sales area.

[0111] In addition, the area detection learning processing unit 201 receives an operation to perform processing to generate an area detection model for the event sales floor on the management terminal 2, extracts image data of the event sales floor and annotation data tagged with the event shelf area 102 from that image data to be stored in the area detection learning data storage unit 200, and uses this to generate a learning model that performs area detection on the image data of the event sales floor.

[0112] The generated learning model for detecting the area of ​​the regular sales area and the learning model for detecting the area of ​​the special sales area are stored in a predetermined storage area.

[0113] Next, the process of generating a decision model will be described.

[0114] The determination image data storage unit 210 stores image data of various regular sales areas, or image data of product display areas and price tag areas cut out from image data of regular sales areas. It also stores image data of various special sales areas, or image data of the special shelf area 102 cut out from image data of the special sales areas.

[0115] When the management terminal 2 receives a command to generate learning image data, the learning image data generation processing unit 211 extracts image data of various regular sales areas, or image data of product display areas and price tag areas cut out from the image data of the regular sales areas, from the determination image data storage unit 210. It also extracts image data of various special sales areas, or image data of the special shelf area 102 cut out from the image data of the special sales areas.

[0116] Then, each extracted image data is processed to improve its image quality, such as by blurring or motion blurring. Furthermore, image data that has undergone image quality processing is annotated with information indicating the image quality corresponding to the processing. For example, if motion blurring is performed, it is annotated as Motion Blurred, if a one-pixel blurring process is performed, it is annotated as Slightly Defocused, and if a three-pixel blurring process is performed, it is annotated as Defocused.

[0117] Any image quality processing can be applied, and information corresponding to the processing can be annotated.

[0118] Furthermore, for image data that has not undergone image quality processing for each extracted image, the learning image data generation processing unit 211 annotates information that indicates that the image quality is free from out-of-focus or blur, such as sharpness.

[0119] Then, annotation data, which is a set of image data and annotations, is stored in the determination learning data storage unit 212.

[0120] The judgment learning processing unit 213 receives an operation to perform processing to generate a judgment model for the standard sales area on the management terminal 2, and extracts image data of the standard sales area to be stored in the judgment learning data storage unit 212, or image data cut out from the image data of the standard sales area of ​​the product display area and price tag area, and annotation data annotated with information indicating the image quality, and uses this to generate a learning model that performs judgment processing on the image data of the standard sales area.

[0121] In addition, the judgment learning processing unit 213 accepts an operation to perform processing to generate a judgment model for the event sales floor on the management terminal 2, extracts image data of the event sales floor, or image data obtained by cutting out a product shelf area from the image data of the event sales floor and annotation data annotated with information indicating its image quality, to be stored in the judgment learning data storage unit 212, and uses this to generate a learning model that performs judgment processing on the image data of the event sales floor.

[0122] The generated learning model for determining whether a regular sales area is a regular sales area and the learning model for determining whether a special sales area is a special sales area are stored in a predetermined storage area.

[0123] A photographer who takes photographs of a store's regular sales area, special sales area, etc. installs a predetermined application program on the photography terminal 3 used in the information processing system 1 of the present invention and makes it available for use before starting the photography work.

[0124] This application program includes a learning model that performs area detection for regular sales areas, generated by the area detection model generation processing unit 20, a learning model that performs area detection for special sales areas, a learning model that performs judgment processing for regular sales areas, generated by the judgment model generation processing unit 21, and a learning model that performs judgment processing for special sales areas, and upon installation as described above, these learning models become available for use on the photography terminal 3.

[0125] When a photographer takes pictures of a regular sales area, a special sales area, etc. in a store using the photography terminal 3 in the information processing system 1 of the present invention, the photographer starts up an application program installed on the photography terminal 3. Then, the photographer selects whether to take pictures of the regular sales area or the special sales area.

[0126] This instruction is accepted by the application program of the photographing terminal 3. If it is accepted that the photograph was taken at a regular sales area, the area detection processing unit 32 and the determination processing unit 34 will input data into the learning model for regular sales areas, and if it is accepted that the photograph was taken at a special sales area, the area detection processing unit 32 and the determination processing unit 34 will input data into the learning model for special sales areas.

[0127] First, the process performed by the photographing terminal 3 when an instruction to photograph a regular sales area is received will be described.

[0128] The photographing processing unit 30 receives an instruction from the photographer to photograph the regular sales area and activates the photographing device in the photographing terminal 3 (S100). When the photographer then points the photographing device of the photographing terminal 3 toward the regular sales area, the photographing device photographs the regular sales area (S110), and the photographing processing unit 30 accepts the image data (for example, image data of 20 frames per second, or a portion of the image data). The photographing processing unit 30 displays the accepted image data on the display device 72 of the photographing terminal 3. An example of the screen at this time is shown in FIG. 8. In FIG. 8, a photograph of a display shelf that is a regular sales area is taken, and guidelines 105 are displayed to make it easier to photograph according to the width of the display shelf so that the photograph can be taken appropriately.

[0129] Therefore, it is preferable for a photographer taking a photo of a standard sales area or a display shelf to take a photo from a position where the left and right edges of the display shelf are aligned with the guide line 105.

[0130] Then, the area detection processing unit 32 detects the product display area and the price tag area by inputting the image data as input values ​​into the area detection model for the regular sales area stored in the area detection model storage unit 31 (S120). An example of a screen showing the detected areas is shown in Fig. 9. Note that the product display area and the price tag area are highlighted on the screen in Fig. 9 for ease of understanding.

[0131] As described above, the area detection processing unit 32 inputs the image data of each frame received from the photography device by the photography processing unit 30 as input values ​​into the area detection model for the regular sales area stored in the area detection model storage unit 31, thereby identifying and displaying the product display area and price tag area in the image data.

[0132] 8 and 9 show the case where the photographing terminal 3 is held vertically to photograph the entire display shelves in the standard sales area, but as shown in Fig. 10, the photographing terminal 3 may also be held horizontally to photograph. In this case, for example, it is possible to take an enlarged photograph of some shelves in the display shelves in the standard sales area. Note that, as with Fig. 9, the product display area and price tag area are highlighted on the screen in Fig. 10 for ease of understanding.

[0133] Then, when the photographer performs an image data recording operation (photographing operation) such as pressing the shutter of the photographing terminal 3 (S130), the photographing processing unit 30 records the image data in a predetermined storage area (S140). In addition, the area detection processing unit 32 identifies the product display area and price tag area detected in the recorded image data.

[0134] Then, the determination processing unit 34 performs a determination process on the image data recorded in S140 to determine whether the image is out of focus, blurred, or the like, using a determination model for a regular sales area stored in the determination model storage unit 33 (S150).

[0135] For example, the judgment processing unit 34 sets predetermined verification areas 101 in the image data recorded in S140, such as near the left and right edges of the product display area or price tag area located on the top shelf, near the left and right edges of the product display area or price tag area located on the bottom shelf, and near the center of the image data, and inputs the image data of each verification area 101 as input values ​​to the judgment model for a regular sales floor stored in the judgment model storage unit 33. This is schematically shown in FIG. 11. In FIG. 11, verification areas 101a and 101b are set near the left and right edges of the product display area on the top shelf, verification areas 101d and 101f are set near the left and right edges of the price tag area on the bottom shelf, and verification area 101c is set near the center of the image data, all with predetermined sizes, and the image data of each of these verification areas 101a to 101e is input to the judgment model for a regular sales floor to determine whether the image quality satisfies predetermined conditions (S160).

[0136] For example, in this determination, image quality determination results such as Sharp, Motion Blurred, Defocused, Slightly Defocused, etc. are obtained for each verification area 101, and if the majority of the verification areas 101 in the image data are determined to have Sharp image quality (S160), it is determined that the image was captured normally, and the image data received in S140 or a portion thereof (for example, image data of the product display area or price tag area) is stored in a predetermined storage area (S170). An example of the screen in this state is shown in Fig. 12.

[0137] If the majority of the verification areas 101 in the image data are determined to have Motion Blurred (blurred) or Defocused (out-of-focus) image quality (S160), it is determined that re-capture should be performed, and the determination processing unit 34 causes the display device 72 of the photographing terminal 3 to display a message urging re-capture (S180). An example of the screen in this state is shown in Fig. 13. Then, the image data accepted in S140 is discarded, and the processes from S110 onwards are repeated again.

[0138] If the majority of the verification areas 101 in the image data are determined to have slightly defocused image quality (S160), the determination processing unit 34 determines to leave it up to the photographer to decide whether to re-shoot, and causes the display device 72 of the photographing terminal 3 to display a message prompting the photographer to decide whether to re-shoot (S190). An example of the screen in this state is shown in FIG. 14. Then, if a selection to store the image data as is (without re-shooting) is accepted from the photographer (S190), the image data accepted in S140 or a part thereof (for example, image data of the product display area or price tag area) is stored in a predetermined storage area. On the other hand, if a selection to re-shoot is accepted, the image data accepted in S140 is discarded, and the processing from S110 onwards is repeated again.

[0139] Then, at a predetermined timing, the transmission processing unit 35 sends the image data stored in the storage area or a part thereof (for example, image data of the product display area or the price tag area) to the management terminal 2 (S200). At this time, information necessary for the analysis process described below, such as the date and time of the image capture, store identification information such as the store name, and information indicating the location of the sales floor where the image was captured, may also be sent in association with the image data.

[0140] Next, the process in the photographing terminal 3 when an instruction to photograph the special sale area is received will be described.

[0141] The photography processing unit 30 receives an instruction from the photographer to photograph the event sales area, and activates the photography device in the photography terminal 3 (S100). When the photographer points the photography device of the photography terminal 3 toward the event sales area, the photography device photographs the event sales area (S110), and the photography processing unit 30 receives the image data (for example, image data of 20 frames per second, or a portion of the image data). The photography processing unit 30 displays the received image data on the display device 72 of the photography terminal 3. FIG. 15 shows an example of the screen at this time. FIG. 15 shows a case where a display shelf in the event sales area is photographed. Unlike regular sales areas, event sales areas generally have an irregular shape, so unlike regular sales areas, it is preferable not to display the guideline 105, although guideline 105 or an area to guide the position may be displayed.

[0142] Then, the area detection processing unit 32 detects the event shelf area 102 by inputting the image data as an input value into the area detection model for the event sales area stored in the area detection model storage unit 31 (S120). An example of a screen showing the detected area is shown in Figure 5(a).

[0143] As described above, the area detection processing unit 32 inputs the image data of each frame received from the photography device by the photography processing unit 30 as input values ​​into the area detection model for the event sales floor stored in the area detection model storage unit 31, thereby identifying and displaying the event shelf area 102 in the image data.

[0144] Then, when the photographer performs a recording operation (photographing operation) of the image data, such as pressing the shutter of the photographing terminal 3 (S130), the photographing processing unit 30 records the image data in a predetermined storage area (S140). In addition, the area detection processing unit 32 identifies the event shelf area 102 detected in the recorded image data.

[0145] Then, the determination processing unit 34 performs a determination process on the image data recorded in S140 to determine whether the image data is out of focus, blurred, or the like, using the determination model for the special sales area stored in the determination model storage unit 33 (S150).

[0146] For example, the judgment processing unit 34 searches for and sets as the verification area 101 a predetermined verification area 101 set in the image data recorded in S140, for example, a predetermined area in the event shelf area 102 in the event sales area that is located farthest from the center of the rectangular area 103 containing the event shelf area 102 in the directions of the upper left and right ends and the lower left and right ends.

[0147] The image data of each set verification area 101 is then input as an input value to a determination model for the special event sales area stored in the determination model storage unit 33. This is shown schematically in Fig. 16. In Fig. 16, verification areas 101a and 101b are set near the upper left and right ends of the special event shelf area 102, verification areas 101d and 101f are set near the lower left and right ends of the special event shelf area 102, and verification area 101c is set near the center of the image data, all with predetermined sizes, and the image data of each of these verification areas 101a to 101e is input to the determination model for the special event sales area, and it is determined whether the image quality satisfies predetermined conditions (S160).

[0148] For example, in this determination, image quality determination results such as Sharp, Motion Blurred, Defocused, Slightly Defocused, etc. are obtained for each verification area 101, and if the majority of the verification areas 101 in the image data are determined to have Sharp image quality (S160), it is determined that the image was captured normally, and the image data accepted in S140 or a portion thereof (for example, image data of the event shelf area 102) is stored in a predetermined storage area (S170). An example of the screen in this state is shown in Fig. 12.

[0149] If the majority of the verification areas 101 in the image data are determined to have Motion Blurred (blurred) or Defocused (out-of-focus) image quality (S160), it is determined that re-capture should be performed, and the determination processing unit 34 causes the display device 72 of the photographing terminal 3 to display a message urging re-capture (S180). An example of the screen in this state is shown in Fig. 13. Then, the image data accepted in S140 is discarded, and the processes from S110 onwards are repeated again.

[0150] If the majority of the verification areas 101 in the image data are determined to have slightly defocused image quality (S160), the determination processing unit 34 determines to leave it up to the photographer to decide whether to re-shoot, and causes the display device 72 of the photographing terminal 3 to display a message prompting the photographer to decide whether to re-shoot (S190). An example of the screen in this state is shown in FIG. 14. Then, if a selection to store the image data as is (not to re-shoot) is accepted from the photographer (S190), the image data accepted in S140 or a part thereof (for example, image data of the event shelf area 102) is stored in a predetermined storage area. On the other hand, if a selection to re-shoot is accepted, the image data accepted in S140 is discarded, and the processing from S110 onwards is repeated again.

[0151] Then, at a predetermined timing, the transmission processing unit 35 sends the image data stored in the storage area or a part thereof (for example, image data of the event shelf area 102) to the management terminal 2 (S200). At this time, information necessary for the analysis process described below, such as the date and time of photography, store identification information such as the store name, and information indicating the location of the sales floor where the photograph was taken, may also be sent in association with the image data.

[0152] If the determination processing unit 34 determines that the image was captured successfully, or if it receives an instruction from the photographer that re-capture is not required, the image data captured by the image capture processing unit 30, or the image data of the product display area, shelf area, and / or event shelf area 102 in the image data, is stored in a predetermined storage area. Then, the transmission processing unit 35 sends the image data stored in the predetermined storage area to the management terminal 2 at a predetermined timing.

[0153] The photographed image data reception processing unit 220 of the management terminal 2 receives the image data of the regular sales floor and / or image data of the product display area or price tag area in the image data of the regular sales floor sent by the photographing terminal 3 in S200, and the image data of the special sales floor and / or image data of the special shelf area 102 in the image data of the special sales floor, and stores them in the photographed image data storage unit 221.

[0154] Then, at a predetermined timing, the identification processing unit 222 extracts image data of the regular sales area and / or image data of the product display area or price tag area in the image data of the regular sales area stored in the photographed image data storage unit 221, image data of the special sales area and / or image data of the special shelf area 102 in the image data of the special sales area, and performs an analysis process to identify the products, prices, etc. in the extracted image data (S210).

[0155] For example, the extracted image data is input as input values ​​into a learning model for analysis processing, and the products shown in the product display area or event shelf area 102, the prices of the price tags shown in the price tag area or event shelf area 102, etc. are identified.

[0156] By performing the above-described processing, image data that has been determined to be free from out-of-focus or blur is used by the analysis processing unit 22 to perform analysis processing of products and prices, making it less likely that errors will occur in the analysis processing in the analysis processing unit 22. Furthermore, since support is provided so that image data that will undergo normal analysis processing is captured at the time of shooting, it is possible to reduce the burden of re-shooting if an error is discovered at the analysis processing stage. [Example]

[0157] In the above-described first embodiment, five regions are used as the verification regions 101 in the determination processing unit 34, but more or fewer regions may be used. Reducing the number of verification regions 101 can reduce the processing time. To reduce the processing time and maintain the accuracy of verification, it is preferable to use about three regions. For example, in addition to providing verification regions 101 near the upper left and right edges and near the right or left edge of the lower edge, verification regions 101 may also be provided near the lower left and right edges and near the left edge of the right branch of the upper edge.

[0158] It is preferable to locate each inspection region 101 near the periphery, and it is advisable to locate it as far away as possible from the center of the image data, because the focus tends to be near the center and often shifts near the periphery.

[0159] Furthermore, when photographing by holding the photographing terminal 3 horizontally as shown in Figure 10, i.e., when photographing from a close distance, only one or two areas of the product display area and price tag area are often photographed, so it is preferable to photograph near the center and near the upper left and right edges, or near the center and near the lower left and right edges, of the product display area and / or price tag area.

[0160] In the above-described first embodiment, the judgment processing unit 34 uses the judgment results of the image quality in the majority of the verification regions 101, but the judgment result of the most common image quality may also be used. Furthermore, the judgment results may be weighted, and weighted so that the judgment results of the verification regions 101 near the periphery are given more importance than the judgment results of the verification regions 101 near the center. [Example]

[0161] As a modification of each of the above-described embodiments, the information processing system 1 in this embodiment may have an imaging condition processing unit 36 ​​that determines whether image data satisfies predetermined imaging conditions in addition to the area detection processing by the area detection processing unit 32 in the imaging terminal 3. An example of the overall processing function of the information processing system 1 in this embodiment is shown in a block diagram in Fig. 17. Also, an example of the overall processing of the information processing system 1 in this embodiment is shown in Fig. 18.

[0162] The photographing condition processing unit 36 ​​judges whether the photographing conditions for the photographing subject are met correctly. That is, it judges whether the entire display shelf to be photographed is included in the image data at a sufficiently large size without protruding from the image data, whether the photographing terminal 3 is facing upward or downward so that the upper or lower end of the photographing subject is not small in the image (downward photographing, upward photographing), whether the photographing position is shifted sideways relative to the shelf and the shelf to be photographed is not distorted (oblique photographing), etc.

[0163] To determine whether the entire display shelf to be photographed fits into the image data at a sufficiently large size without extending beyond the image data, the photographing condition processing unit 36 ​​detects the breaks in the display shelf from the image data and determines whether the breaks are within the range of the guideline 105 in the image data, or whether the width direction of the product display area and price tag area detected by the area detection processing unit 32 is within the range of the guideline 105. Then, by determining whether the width direction of the product display area and price tag area detected by the area detection processing unit 32 is at a certain ratio or greater to the width direction length of the guideline 105, it is determined whether the image is at a predetermined size or greater. This is schematically shown in Figure 19. In Figure 19, the shaded area indicates that it does not fit within the range of the guideline 105.

[0164] Furthermore, whether the photographing terminal 3 is facing upward or downward, causing the upper and lower edges of the photographed subject to appear small, can be determined by using a gyro sensor in the photographing terminal 3 to determine whether the photographing direction is upward or downward, or by determining, for example, whether the left and right edges of the product display area and price tag area detected by the area detection processing unit 32 become narrower as they move downward (downward photographing) or upward (upward photographing). Also, deviation of the photographing position relative to the display shelf can be determined by whether the left and right edges of the product display area and price tag area detected by the area detection processing unit 32 are inward as they move downward. Figure 20 shows an example of a warning displayed when "oblique photographing" is performed.

[0165] The imaging condition processing unit 36 ​​may display an alert if any of the above conditions is not met.

[0166] It is preferable that the photographing processing unit 30 executes control processing so that photographing operations (image data recording operations), such as pressing the photographing button, cannot be performed until the photographing condition processing unit 36 ​​determines that the conditions are satisfied. Then, as shown in Fig. 21, when the photographing condition processing unit 36 ​​determines that all conditions are satisfied, the photographing processing unit 30 executes control processing so that photographing operations, such as pressing the photographing button, can be performed.

[0167] Next, an example of processing by the information processing system 1 in this embodiment will be described with reference to the flowchart of FIG.

[0168] The process of generating the area detection model and the process of generating the judgment model are the same as those in the first embodiment, and therefore the description thereof will be omitted.

[0169] As in the first embodiment, an application program is installed on the photographing terminal 3. A learning model for performing area detection and a learning model for performing determination processing are also available on the photographing terminal 3.

[0170] When a photographer takes pictures of a regular sales area, a special sales area, etc. in a store using the photography terminal 3 in the information processing system 1 of the present invention, the photographer starts up an application program installed on the photography terminal 3. Then, the photographer selects whether to take pictures of the regular sales area or the special sales area.

[0171] This instruction is accepted by the application program of the photographing terminal 3. If it is accepted that the photograph was taken at a regular sales area, the area detection processing unit 32 and the determination processing unit 34 will input data into the learning model for regular sales areas, and if it is accepted that the photograph was taken at a special sales area, the area detection processing unit 32 and the determination processing unit 34 will input data into the learning model for special sales areas.

[0172] First, the process performed by the photographing terminal 3 when an instruction to photograph a regular sales area is received will be described.

[0173] The photographing processing unit 30 receives an instruction from the photographer to photograph the regular sales area and activates the photographing device in the photographing terminal 3 (S300). Then, when the photographer points the photographing device of the photographing terminal 3 toward the regular sales area, the photographing device photographs the regular sales area (S310), and the photographing processing unit 30 receives the image data (for example, image data of 20 frames per second, or a portion of the image data). The photographing processing unit 30 displays the received image data on the display device 72 of the photographing terminal 3.

[0174] Therefore, it is preferable for a photographer taking a photo of a standard sales area or a display shelf to take a photo from a position where the left and right edges of the display shelf are aligned with the guide line 105.

[0175] The area detection processing unit 32 then inputs the image data as input values ​​into the area detection model for the regular sales area stored in the area detection model storage unit 31, thereby detecting the product display area and the price tag area (S320).

[0176] As described above, the area detection processing unit 32 inputs the image data of each frame received from the photography device by the photography processing unit 30 as input values ​​into the area detection model for the regular sales area stored in the area detection model storage unit 31, thereby identifying and displaying the product display area and price tag area in the image data.

[0177] Then, the photographing condition processing unit 36 ​​determines whether the photographed image data satisfies predetermined conditions (S330).

[0178] The photographing condition processing unit 36 ​​detects the breaks in the display shelves from the image data and determines whether the breaks are within the range of the guideline 105 of the image data, or whether the width direction of the product display area and price tag area detected by the area detection processing unit 32 is within the range of the guideline 105. Then, by determining whether the width direction of the product display area and price tag area detected by the area detection processing unit 32 is at a certain ratio or more to the width direction length of the guideline 105, it is determined whether the image is at a predetermined size or larger.

[0179] If these conditions are not met, a warning is issued to the photographer about the areas that have not been properly photographed, for example by shading the product display area or price tag area that does not meet the conditions in a predetermined color (S340), as shown in Fig. 19. It is preferable to change the display of the areas that have not been properly photographed, such as product display areas or price tag areas that do not meet the conditions, by shading them, highlighting them, or surrounding them with a frame, so that the photographer can see the areas that have not been properly photographed.

[0180] The photographing condition processing unit 36 ​​also determines whether the photographing orientation of the photographing subject is correct. For example, a sensor such as a gyro sensor in the photographing terminal 3 is used to determine whether the photographing orientation is upward or downward, and also determines whether the photographing orientation in the up-down direction is correct by determining whether the left and right edges of the product display area and price tag area detected by the area detection processing unit 32 become narrower as they go downward (downward photographing) or upward (upward photographing). Furthermore, the area detection processing unit 32 determines whether the photographing is oblique by determining whether the left and right edges of the product display area and price tag area detected by the area detection processing unit 32 move inward as they go downward. A warning when the photographing is oblique is shown in FIG. 20.

[0181] As described above, the shooting condition processing unit 36 ​​determines whether the shooting conditions are met for the image to be shot by the shooting processing unit 30, and if the conditions are not met, displays a warning, and if the conditions are met, the image can be shot.

[0182] Then, when the photographer performs a recording operation (photographing operation) of image data, such as pressing the shutter of the photographing terminal 3 (S350), the photographing processing unit 30 records the image data in a predetermined storage area (S360). In addition, the area detection processing unit 32 identifies the product display area and price tag area detected in the recorded image data.

[0183] The subsequent processing is the same as in the first embodiment, and therefore a description thereof will be omitted.

[0184] The photographing condition determination process and warning display process by the photographing condition processing unit 36 ​​can be applied not only to photographing a regular sales area but also to photographing a special sales area.

[0185] For example, by determining whether the width of the event area is within the range of the guideline 105 and has a ratio of a certain value or more to the width of the guideline 105, it is possible to determine whether the image data does not extend beyond the range and is captured at a size greater than a predetermined value.

[0186] Furthermore, a sensor such as a gyro sensor can be used to detect whether the photographing terminal 3 is facing upward or downward, and determine whether the photographing terminal 3 is facing upward or downward. [Example]

[0187] In each of the above-described embodiments, the order of the processes can be changed as appropriate, and some processes can be omitted or added. For example, a correction process can be performed to correct the image data captured by the image capture processing unit 30 so that it is captured from a position facing the camera. When this correction process is performed by the image capture terminal 3, it can be performed at any timing as long as it is before the transmission processing unit 35 corrects the image data. Furthermore, the correction process can be performed before the identification process by the identification processing unit 222 in the analysis processing unit 22 of the management terminal 2.

[0188] In the above-described embodiments, the case where the area detection model and the judgment model are operated in the photographing terminal 3 has been described, but the management terminal 2 may have an area detection model storage unit 31, an area detection processing unit 32, a judgment model storage unit 33, and a judgment processing unit 34. In this case, image data photographed by the photographing processing unit 30 may be sent to the management terminal 2, where area detection processing and judgment processing may be performed in the management terminal 2, and the results may be returned to the photographing terminal 3. Furthermore, the management terminal 2 may be provided with either an area detection model for a regular sales area or an area detection model for a special sales area, or either a judgment model for a regular sales area or a judgment model for a special sales area, and some of the processing may be executed by the management terminal 2.

[0189] In each of the above-described embodiments, the image quality determination process uses four levels of image quality, such as Sharp, Motion Blurred, Defocused, and Slightly Defocused, but the number of image quality levels may be more or less than four.

[0190] In the above description, the judgment processing unit 34 sets the verification area 101 near the left, right, top, bottom, and center, but if a specific object or a specific area is detected, that area can also be set as the verification area 101.

[0191] For example, in the case of image data for a standard sales area, in addition to the product display area and / or price tag area, individual price tags (price cards) are trained as detection targets, and the area detection model of the area detection processing unit 32 is made to detect the areas of the individual price tags in addition to the product display area and / or price tag area. A part or all of the areas of the individual price tags may be set as the verification area 101.

[0192] Furthermore, the photographer may input the setting of the area to be used as the verification area 101, and this may be accepted.

[0193] Furthermore, it is preferable that the entire verification area 101 is included in the product display area, price tag area, and event area, but a part of the verification area 101 may be included in the product display area, price tag area, and event area.

[0194] Although the photographing terminal 3 is described as a terminal used by a photographer to take photographs, other cameras such as security cameras in stores can also be used as the photographing terminal 3. [Industrial Applicability]

[0195] By using the information processing system 1 of the present invention, it is possible to assist the photographer in taking pictures of display shelves displaying products, so that the photographer can capture image information that is not out of focus or blurred, which would hinder image recognition processing. [Explanation of symbols]

[0196] 1: Information processing system 2: Management terminal 3: Shooting device 20: Region detection model generation processing unit 21: Decision model generation processing unit 22: Analysis processing unit 30: Shooting processing unit 31: Region detection model storage unit 32: Area detection processing unit 33: Decision model memory unit 34: Judgment processing unit 35: Transmission processing unit 36: Shooting condition processing section 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74:Communication equipment 101: Verification area 102: Event shelf area 103: Rectangular area including the event shelf area 104: Line segment tangent to the event shelf area 105: Guidelines 200: Region detection learning data storage unit 201: Area detection learning processing unit 210: Determination image data storage unit 211: Learning image data generation processing unit 212: Judgment learning data storage unit 213: Decision learning processing unit 220: Photographed image data reception processing unit 221: Photographed image data storage unit 222: Identification processing unit

Claims

1. An information processing system that supports photographing of a product sales floor, The information processing system includes: an image capturing processing unit that captures an image of the sales floor using an image capturing device of an image capturing terminal; an area detection processing unit that detects a predetermined area from image data captured by the image capturing processing unit; a determination processing unit that performs a determination process of image quality in a part or all of the detected area; An information processing system comprising:

2. The determination processing unit a verification area to be subjected to the image quality determination process is set in the detected area; If the image quality of the verification area satisfies a predetermined condition, the image data is stored as image data to be processed.

2. The information processing system according to claim 1, wherein:

3. The determination processing unit If the image quality of the verification area does not satisfy the predetermined condition, the image data is prompted to be re-captured or re-captured.

3. The information processing system according to claim 2.

4. The determination processing unit If the image quality of the verification area does not satisfy the predetermined condition, a message is displayed prompting the user to select whether to store the image data as image data to be processed or to take a re-image, and the user is then prompted to select the option.

4. The information processing system according to claim 3.

5. The determination processing unit A plurality of the verification regions are set; determining the image quality for each of the plurality of verification regions; modifying the processing of the image data according to the number of image qualities determined for each verification region; 3. The information processing system according to claim 2.

6. The determination processing unit determining, for each of the plurality of set verification regions, one or more of normal, blurred, out of focus, and slightly out of focus as the image quality; If the number of the verification areas determined to have normal image quality is a majority or a large number, the image data is stored as image data to be processed.

6. The information processing system according to claim 5.

7. The determination processing unit determining, for each of the plurality of set verification regions, one or more of normal, blurred, out of focus, and slightly out of focus as the image quality; If the majority or a large number of the verification areas are determined to have blurred or out-of-focus image quality, the image data is prompted to be re-captured or re-captured.

6. The information processing system according to claim 5.

8. The determination processing unit determining, for each of the plurality of set verification regions, one or more of normal, blurred, out of focus, and slightly out of focus as the image quality; If the majority or majority of the verification areas are determined to have slightly out-of-focus image quality, a message is displayed prompting the user to select whether to store the image data as image data to be processed or to take a re-image, and the user is then prompted to select the option.

6. The information processing system according to claim 5.

9. The determination processing unit The verification area is set to one or more of the vicinity of the upper, lower, left, and right ends of the area detected by the area detection processing unit, and the vicinity of the center.

3. The information processing system according to claim 2.

10. The information processing system includes: The photographing processing unit has an area detection learning processing unit that generates a learning model for detecting a predetermined area from the photographed image data, the area detection model generated by the area detection learning processing unit functions on the photographing terminal; The image data captured by the image capturing processing unit is input to the area detection model to detect the predetermined area.

2. The information processing system according to claim 1, wherein:

11. The information processing system includes: a judgment model generation processing unit that generates a judgment learning model that determines whether the image data captured by the image capture processing unit or a part thereof satisfies image quality conditions to an analyzable extent, the determination model generated by the determination learning processing unit functions on the photographing terminal; a part or all of the image data captured by the image capture processing unit is input to the determination model to determine the image quality conditions; 2. The information processing system according to claim 1, wherein:

12. The information processing system includes: an imaging condition processing unit that determines whether imaging conditions are met and displays a warning when it is determined that the imaging conditions are not met; 2. The information processing system according to claim 1, further comprising:

13. The imaging condition processing unit Determine whether the subject is large enough to fit into the image data without overflowing, or whether the subject is facing the camera.

13. The information processing system according to claim 12.

14. The imaging condition processing unit Using the relationship between the position of the gap in the display shelf detected from the image data or the width of the area detected by the area detection processing unit and the width of the guideline of the image data, it is determined whether the object to be photographed is contained in the image data at a sufficiently large size without protruding from the image data.

14. The information processing system according to claim 13.

15. The imaging condition processing unit determining whether the position of the gap in the display shelf detected from the image data or the position of the area detected by the area detection processing unit is located within a predetermined range of the image data; and determining whether the width of the area detected by the area detection processing unit is equal to or greater than a certain ratio to the width of the guideline; 15. The information processing system according to claim 14.

16. The imaging condition processing unit determining the photographing direction using a sensor provided in the photographing terminal, or determining the photographing direction using the positional relationship of the edges of the area detected by the area detection processing unit; 14. The information processing system according to claim 13.

17. The imaging condition processing unit If it is determined that the conditions regarding photography are not satisfied, a warning is displayed by changing the display of the area in which the conditions regarding photography are not satisfied.

13. The information processing system according to claim 12.

18. The imaging condition processing unit If it is determined that the photographing conditions are not met, the image data cannot be recorded.

13. The information processing system according to claim 12.

19. The information processing system includes: a captured image data reception processing unit that receives the image data or a part thereof that has been determined to be the data to be processed by the determination processing unit; an identification processing unit that identifies one or more of product identification information and price from the received image data or a portion thereof; 2. The information processing system according to claim 1, further comprising:

20. A photographing terminal used when photographing a product sales floor, The photographing terminal an image capturing processing unit that captures an image of the sales floor using an image capturing device of an image capturing terminal; an area detection processing unit that detects a predetermined area from image data captured by the image capturing processing unit; a determination processing unit that performs a determination process of image quality in a part or all of the detected area; An imaging terminal comprising:

21. The photographing terminal an area detection model storage unit that stores a learning model for detecting a predetermined area from the image data; The area detection learning processing unit The image data captured by the photographing processing unit is input to the area detection model to detect the predetermined area, and the area is displayed on a display device of the photographing terminal. The photographing terminal according to claim 20.

22. The photographing terminal a judgment model storage unit that stores a judgment learning model that determines whether the image data or a part thereof satisfies a predetermined image quality condition, The determination processing unit a part or all of the image data captured by the image capture processing unit is input to the determination model to determine the image quality conditions; The photographing terminal according to claim 20.

23. The photographing terminal a photographing condition processing unit that determines whether the image data satisfies a photographing condition, and displays a warning on a display device of the photographing terminal when it is determined that the image data does not satisfy the photographing condition; 21. The photographing terminal according to claim 20, further comprising:

24. Computer, an image capturing processing unit that captures an image of the product sales floor using an image capturing device of the image capturing terminal; an area detection processing unit that detects a predetermined area from the image data captured by the image capturing processing unit; a determination processing unit that performs a determination process of image quality in a part or all of the detected area; An information processing program characterized by causing the program to function as:

25. The photographing device used to photograph the product sales floor is an image capturing processing unit that captures an image of the sales floor using an image capturing device of an image capturing terminal; an area detection processing unit that detects a predetermined area from the image data captured by the image capturing processing unit; a determination processing unit that performs a determination process of image quality in a part or all of the detected area; A program characterized by functioning as

Images