Information processing system

Abstract

To provide an information processing system that supports a photographer in taking pictures of a display shelf displaying merchandise, such that the photographer can take an image without blurring or blurring that may cause problems in image recognition processing.SOLUTION: An information processing system that supports photographing of a product sales floor includes an imaging processing unit that photographs the sales floor with a photographing device of a photographing terminal, an area detection processing unit that detects a specified area from image data photographed by the imaging processing unit, and a determination processing unit that performs image quality determination processing for part or all of the detected area.SELECTED DRAWING: Figure 1

Description

Technical Field

[0004] , , , , , , , , , , , , , ,

[0005] ,

[0003]

[0001] The present invention has invented an information processing system that supports a photographer's shooting so that an image without blurring or defocus that may interfere with image recognition processing can be captured when shooting a display shelf or the like for displaying products.

Background Art

[0002] In various stores such as convenience stores and supermarkets, it is common to sell the products being sold by placing them on display shelves. Therefore, by arranging a plurality of products on the display shelf, even if one product is purchased, the same product can be purchased by another person. In addition, a competitive relationship of superiority and inferiority occurs, where products arranged in a large number at prominent positions are noticed and sold more than other products. Therefore, it is important in the sales strategy of products to manage where and how many products are displayed on the display shelf and the prices of those products.

[0003] Therefore, by photographing the display shelf with a photographing device such as a camera and automatically recognizing the objects shown in the image information, the display position and number of products, the products being displayed, and the prices described on the price tags (price cards) are grasped. Therefore, there are the technologies disclosed in Patent Documents 1 and 2 below.

[0004] In recent years, as a photographing device for photographing a display shelf, a camera or the like mounted on a portable communication terminal such as a smartphone may be used. An example of this photographing device is disclosed in Patent Document 3 below.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Patent Document 3

[0006] Patent Document 3 describes a system that involves attaching codes or other labels to display shelves in advance, and then using a smartphone to photograph the shelves and determine the location information of the price tags attached to them. However, the system in Patent Document 3 requires that codes or other labels be attached to the display shelves beforehand. Therefore, it cannot be applied when codes or other labels are not attached to the display shelves.

[0007] Furthermore, as described in Patent Documents 1 and 2, in order to perform product recognition using image information of display shelves, high-quality images free from blur or out-of-focus shots are required. Therefore, when photographing display shelves, it is necessary to take pictures without blur or out-of-focus shots. While obvious blur or out-of-focus shots can be detected at the time of shooting, slight blur or out-of-focus shots will only be discovered when performing product recognition from the image information of the display shelves. If blur or out-of-focus shots are detected during product recognition, it becomes necessary to go back to the store and retake the photos of the display shelves.

[0008] However, re-photographing the display shelves is a significant burden, and the display conditions and prices may change from the time of the initial photo shoot, which is undesirable from the perspective of understanding the display conditions and prices.

[0009] Therefore, when photographing display shelves or similar items, it would be beneficial to be able to warn the photographer at the time of shooting if the image is out of focus or blurry, which could hinder image recognition processing. However, such a system does not currently exist. [Means for solving the problem]

[0010] In view of the above problems, the inventors have invented an information processing system to assist photographers in taking pictures of display shelves and other surfaces where products are displayed, so that they can capture images that are free from blur or smudges that would hinder image recognition processing.

[0011] The first invention is an information processing system that assists in photographing a product sales area, the information processing system comprising a photography processing unit that photographs the sales area with a photography device of a photography terminal, The aforementioned imaging processing unit has a learning model that detects a predetermined region from the image data it has captured, Image data captured by the aforementioned shooting processing unit By entering, A region detection processing unit that detects a predetermined region, The determination model, which determines whether the image data satisfies the image quality conditions to a degree that allows for analysis, is input with image data of a verification region that is the target of the image quality determination process, which is set in part or all of a predetermined region detected by the region detection processing unit. This is an information processing system having a determination processing unit that performs image quality determination processing.

[0012] By configuring the device as described in this invention, it is possible to assist the photographer in taking pictures of display shelves and other surfaces where products are displayed, so that they can capture images that are free from blur and out of focus.

[0013] In the above-described invention, the determination processing unit teeth, The system can be configured to store the image data as the image data to be processed if the image quality of the verification area satisfies predetermined conditions.

[0014] In the above-described invention, the determination processing unit can be configured as an information processing system that prompts or causes the image data to be re-captured if the image quality of the verification area does not satisfy the predetermined conditions.

[0015] As in the present invention, by determining the image quality before the analysis processing in the analysis processing unit, the image quality of the image data to be processed can be maintained at a certain level or higher. Therefore, the amount of image data that needs to be re-captured can be reduced, thereby alleviating the burden of re-capture.

[0016] In the above-described invention, the determination processing unit can be configured as an information processing system that, if the image quality of the verification area does not satisfy the predetermined conditions, further displays a message prompting the user to choose whether to store the image data as image data to be processed or to re-shoot, and accepts such instruction.

[0017] As in the present invention, it may be configured to accept the selection by the photographer as to whether to use the image data to be processed or to perform reshooting. In the case of image quality that is delicate and may or may not be analyzable, it can be decided based on the judgment of the photographer.

[0018] In the above-described invention, the determination processing unit may be configured as an information processing system that sets a plurality of verification regions, determines the image quality for each of the plurality of verification regions, and changes the processing for the image data according to the number of the determined image qualities for each verification region.

[0019] In the above-described invention, the determination processing unit may be configured as an information processing system that determines, for each of the plurality of set verification regions, any one or more of normal, blurry, defocused, and slightly defocused as the image quality, and stores the image data as image data to be processed if the number of verification regions determined to have normal image quality is a majority or a large number.

[0020] In the above-described invention, the determination processing unit may be configured as an information processing system that determines, for each of the plurality of set verification regions, any one or more of normal, blurry, defocused, and slightly defocused as the image quality, and prompts or causes reshooting of the image data if the number of verification regions determined to have blurry or defocused image quality is a majority or a large number.

[0021] In the above-described invention, the determination processing unit may be configured as an information processing system that determines, for each of the plurality of set verification regions, any one or more of normal, blurry, defocused, and slightly defocused as the image quality, displays a message prompting a selection as to whether to store the image data as image data to be processed or to perform reshooting if the number of verification regions determined to have slightly defocused image quality is a majority or a large number, and accepts the instruction.

[0022] The image quality determination process may be configured as in these inventions.

[0023] In the above inventions, the determination processing unit can be configured as an information processing system that sets the verification area to one or more of the vicinity of the upper, lower, left, and right ends and the vicinity of the center of the area detected by the area detection processing unit.

[0024] An image captured by a photographing device generally tends to be in focus near the center, and as it moves away from the center, defocusing, blurring, etc. are more likely to occur, and their effects also become greater. Therefore, by configuring as in the present invention, defocusing, blurring, etc. of the image portion away from the center can be detected and used for image quality determination.

[0025] In the above inventions, the information processing system has a region detection learning processing unit that generates a learning model for detecting a predetermined region from the image data captured by the photographing processing unit. The region detection model generated by the region detection learning processing unit functions in the photographing terminal, and by inputting the image data captured by the photographing processing unit into the region detection model, the predetermined region can be detected. It can be configured as an information processing system.

[0026] In the above inventions, the information processing system determines whether the image data captured by the photographing processing unit or a part thereof satisfies the image quality conditions to an analyzable extent. Decision model It has a determination model generation processing unit that generates, and the determination Model generation The determination model generated by the processing unit functions in the photographing terminal, and by inputting the above-mentioned, into the determination model, the image quality conditions can be determined. It can be configured as an information processing system. Image data of the verification area

[0027] By storing the learning model in the photographing terminal as in these inventions, communication delays can be eliminated and the processing speed can be increased.

[0028] ​In the above-described invention, the information processing system can be configured as an information processing system having a shooting condition processing unit that determines whether the shooting conditions are met and displays a warning if it is determined that the shooting conditions are not met.

[0029] As in the present invention, by displaying a warning when the shooting conditions are not met, the photographer can take pictures under appropriate conditions so that the warning is not displayed.

[0030] In the above-described invention, the shooting condition processing unit can be configured as an information processing system that determines whether the object to be photographed is contained within the image data at a sufficiently large size without extending beyond the image, or the orientation of the object to be photographed.

[0031] In the above-described invention, the shooting condition processing unit can be configured as an information processing system that determines whether the object to be photographed is contained within the image data at a sufficiently large size without extending beyond it, using the relationship between the position of the shelf divider detected from the image data, or the length in the width direction of the region detected by the region detection processing unit, and the width of the guideline in the image data.

[0032] In the above-described invention, the shooting condition processing unit can be configured as an information processing system that determines whether the position of the gap in the display shelf detected from the image data, or the position of the region detected by the region detection processing unit, is located within a predetermined range of the image data, and whether the width of the region detected by the region detection processing unit is in a ratio greater than or equal to a certain length of the width of the guideline.

[0033] In the above-described invention, the shooting condition processing unit can be configured as an information processing system that determines the shooting direction using a sensor provided on the shooting terminal, or by determining the shooting direction using the positional relationship of the edges of the region detected by the region detection processing unit.

[0034] Whether the conditions for photography are met can be achieved by processing as described in these inventions.

[0035] In the above-described invention, the shooting condition processing unit can be configured to display a warning by changing the display of the area where the shooting conditions are not met, if it determines that the shooting conditions are not met, as an information processing system.

[0036] If the conditions for shooting are not met, the photographer can easily recognize the area that does not meet the conditions by changing the display of that area, as in the present invention.

[0037] In the above invention, the shooting condition processing unit is the shooting Regarding It can be configured like an information processing system where image data cannot be recorded if it is determined that the conditions are not met.

[0038] If the shooting conditions are not met, the image data will be wasted. Therefore, it is preferable to be able to control the system so that image data cannot be recorded in the first place.

[0039] In the above invention, the information processing system, in the determination processing unit, Image quality determination process The information processing system can be configured to include: a shooting image data receiving processing unit that receives the target image data or a part thereof; and an identification processing unit that identifies one or more of the following from the received image data or a part thereof: product identification information, price, etc.

[0040] Since the camera accepts image data of a certain quality or higher, the accuracy of the analysis process is improved.

[0041] The 20th invention is a photographing terminal used when photographing a product sales area, wherein the photographing terminal includes a photographing processing unit that photographs the sales area with the photographing device of the photographing terminal, The aforementioned imaging processing unit has a learning model that detects a predetermined region from the image data it has captured, Image data captured by the aforementioned shooting processing unit By entering, A region detection processing unit that detects a predetermined region, The determination model, which determines whether the image data satisfies the image quality conditions to a degree that allows for analysis, is input with image data of a verification region that is the target of the image quality determination process, which is set in part or all of a predetermined region detected by the region detection processing unit. This is a shooting terminal having a determination processing unit that performs image quality determination processing.

[0042] In the above invention, the shooting terminal is a learning model that detects a predetermined region from the image data. Region detection model It has a region detection model storage unit that stores the above Region detection processing unit The system can be configured as a shooting terminal, which inputs the image data captured by the shooting processing unit into the region detection model to detect the predetermined region and displays that region on the display device of the shooting terminal.

[0043] In the invention described above, the imaging terminal is the Image data of the verification area However, it determines whether the predetermined image quality conditions are met. Decision model The determination processing unit has a determination model storage unit that stores the Image data of the verification area By inputting this into the judgment model, it can be configured as a shooting terminal that determines the conditions for image quality.

[0044] In the above-described invention, the shooting terminal may be configured to include a shooting condition processing unit that determines whether the image data satisfies the conditions for shooting, and if it determines that the conditions for shooting are not satisfied, displays a warning on the display device of the shooting terminal.

[0045] The processing of the present invention is preferably performed on a camera terminal.

[0046] The information processing system of the first invention can be realized by executing the program of the present invention on a computer. That is, the computer is used as the shooting processing unit that takes pictures of the product sales area with the shooting device of the shooting terminal, The aforementioned imaging processing unit has a learning model that detects a predetermined region from the image data it has captured, Image data captured by the aforementioned shooting processing unit By entering, Region detection processing unit that detects a predetermined region, The determination model, which determines whether the image data satisfies the image quality conditions to a degree that allows for analysis, is input with image data of a verification region that is the target of the image quality determination process, which is set in part or all of a predetermined region detected by the region detection processing unit.This is an information processing program that functions as a judgment processing unit that performs image quality determination.

[0047] The 20th invention, the photographic terminal, can be realized by executing the program of the present invention on a computer. That is, the photographic terminal used when photographing a product sales area is a photographic processing unit that photographs the sales area with the photographic device of the photographic terminal, The aforementioned imaging processing unit has a learning model that detects a predetermined region from the image data it has captured, Image data captured by the aforementioned shooting processing unit By entering, Region detection processing unit that detects a predetermined region, The determination model, which determines whether the image data satisfies the image quality conditions to a degree that allows for analysis, is input with image data of a verification region that is the target of the image quality determination process, which is set in part or all of a predetermined region detected by the region detection processing unit. This program functions as a judgment processing unit that performs image quality determination. [Effects of the Invention]

[0048] By using the information processing system of the present invention, it becomes possible to assist photographers in taking pictures of display shelves and other surfaces where products are displayed, so that they can capture images that are free from blur or smudges that would hinder image recognition processing. [Brief explanation of the drawing]

[0049] [Figure 1] This is a schematic block diagram showing an example of the configuration of the information processing system of the present invention. [Figure 2] This is a schematic block diagram showing an example of the hardware configuration of a computer used in the information processing system of the present invention. [Figure 3] This flowchart shows an example of the overall processing in the information processing system of the present invention. [Figure 4] This diagram schematically shows the processing overview in the judgment processing unit for a standard sales area. [Figure 5] This diagram schematically shows the processing outline of the judgment processing unit in the case of a special event sales area. [Figure 6] This diagram schematically shows the verification area. [Figure 7] This diagram schematically illustrates another example of a process for exploring the verification area. [Figure 8] This diagram shows an example of a screen when photographing a standard sales area. [Figure 9] This figure shows an example of a screen after area detection processing for the product display area and price tag area has been performed from the screen shown in Figure 8. [Figure 10] This figure shows an example of a screen when the camera is held horizontally. [Figure 11] This diagram schematically shows the validation region that is input as an input value to the standard sales area determination model. [Figure 12] This figure shows an example of the screen displayed when the processing unit determines that the image was captured successfully. [Figure 13] This figure shows an example of the screen displayed when the judgment processing unit determines that reshooting should be performed. [Figure 14] This figure shows an example of a screen displayed when the processing unit determines that it is up to the photographer to decide whether to retake the image. [Figure 15] This diagram shows an example of a screen when taking photos of a promotional sales area. [Figure 16] This diagram schematically shows the validation region that will be input as an input value to the judgment model for event sales areas. [Figure 17] This is a schematic block diagram showing an example of the configuration of the information processing system in Example 3. [Figure 18] This is a flowchart illustrating an example of the overall processing in the information processing system in Example 3. [Figure 19] This figure shows an example of a screen used to determine whether the entire display shelf being photographed is captured at a sufficiently large size and does not extend beyond the image data. [Figure 20] This figure shows an example of a screen displaying a warning when "shooting at an angle." [Figure 21] This figure shows an example of the screen displayed when the shooting conditions processing unit determines that all conditions have been met. [Modes for carrying out the invention]

[0050] Figure 1 shows a block diagram illustrating an example of the overall processing functions of the information processing system 1 of the present invention. The information processing system 1 uses a management terminal 2 and a shooting terminal 3. The management terminal 2 is a computer used by an organization such as a company that operates the information processing system 1. The shooting terminal 3 is a terminal operated by a photographer who takes pictures of product display areas, etc.

[0051] In this invention, a display area refers to a place where goods are displayed, and mainly consists of a regular sales area and a special event sales area. A regular sales area is a place where goods are displayed by placing or hanging items on racks (shelves) of a standard size. A special event sales area is a place where goods are displayed by stacking them in a pyramid shape, or by placing or hanging items in cardboard boxes or baskets. Regular sales areas are often, but are not limited to, set up along the walls of a store or arranged regularly in a predetermined direction within the store. They may also be set up near the cash register. Special event sales areas are often, but are not limited to, set up near the long end of a regular sales area where baskets or cardboard boxes are placed in the aisle and multiple racks are arranged regularly.

[0052] The management terminal 2 and the shooting terminal 3 in the information processing system 1 are implemented using a computer. Figure 2 schematically shows an example of the computer's hardware configuration. The computer has an arithmetic unit 70 such as a CPU that executes program calculations, a storage device 71 such as RAM or a hard disk that stores information, a display device 72 such as a display that shows information, an input device 73 such as a keyboard or mouse that can input information, and a communication device 74 that sends and receives the processing results of the arithmetic unit 70 and the information stored in the storage device 71 via a network such as the Internet or a LAN.

[0053] If the computer is equipped with a touch panel display, the display device 72 and the input device 73 may be configured as an integrated unit. Touch panel displays are often used in portable communication terminals such as tablet computers and smartphones, but are not limited to these.

[0054] A touch panel display is a device in which the functions of a display device 72 and an input device 73 are integrated, as input can be performed directly on the display using a predetermined input device (such as a touch panel pen) or a finger.

[0055] The shooting terminal 3 may be equipped with a camera or other shooting device in addition to the devices described above. A portable communication terminal such as a mobile phone, smartphone, or tablet computer can also be used as the shooting terminal 3. The shooting terminal 3 photographs a predetermined subject and inputs the image data of the subject (photographed image data) into the management terminal 2. For example, it photographs a display shelf in a store and inputs the image data of the display shelf (photographed image data) into the management terminal 2. In this case, products are displayed on the display shelf, and the display shelf is photographed.

[0056] Each of the means in this invention may be physically or in practice the same domain, even though their functions are logically distinct. The processing in each of the means in this invention can be changed in any order as appropriate. Furthermore, some of the processing may be omitted. For example, the orthogonalization process described later can be omitted. In that case, processing can be performed on image data that has not undergone orthogonalization.

[0057] The management terminal 2 of the information processing system 1 has a region detection model generation processing unit 20, a judgment model generation processing unit 21, and an analysis processing unit 22.

[0058] The region detection model generation processing unit 20 generates a machine learning model for region detection from image data captured by the camera terminal 3. The regions targeted for machine learning are the region where products are displayed (product display region), the region where price tags are attached (price tag region), and the region where event shelves are set up (event shelf region 102). The product display region includes, for example, the shelf tiers of display shelves in a regular sales area, and the hanging product region (the region where products such as toothbrushes are hung and displayed). The price tag region includes, for example, the price tags attached to the shelf tiers of display shelves in a regular sales area, or the region where price tags can be attached (the front region of the shelf tier). The event shelf region 102 includes, for example, the region where products are displayed in an event sales area, and the region where product price tags are attached. Unlike the product display area and price tag area of ​​a regular sales area, the event shelf area 102 is generally irregular in shape. Therefore, it is preferable to detect the event shelf area 102 as a combined event shelf area 102 rather than distinguishing it into a product display area where products are displayed and a price tag area where price tags are placed. However, it is also acceptable to detect the area by distinguishing it from the product display area and price tag area.

[0059] As described later, when the regular sales area or event sales area is photographed with the camera device of the shooting terminal 3, the captured image data is input to the learning model (network) generated by the region detection model generation processing unit 20, which then detects various regions such as the product display area, price tag area, and event shelf area 102 that are captured in the image data. This learning model for region detection is generated by the region detection model generation processing unit 20.

[0060] In the region detection model generation processing unit 20, it is preferable to use deep learning as the machine learning method. In this case, the learning model is a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many hidden layers are optimized. Note that the machine learning method used by the region detection model generation processing unit 20 is not limited to deep learning; other machine learning methods can also be used.

[0061] The region detection model generation processing unit 20 includes a region detection learning data storage unit 200 and a region detection learning processing unit 201.

[0062] The region detection learning data storage unit 200 stores learning data (region detection learning data) for machine learning by the region detection learning processing unit 201, which will be described later. The learning data used in this case consists of image data taken of various regular sales areas and event sales areas, and annotation data in which product display areas and price tag areas are tagged in that image data.

[0063] The region detection learning processing unit 201 uses the learning data stored in the region detection learning data storage unit 200 as the correct answer to train a learning model in which the weight coefficients between neurons in each layer of a neural network consisting of many hidden layers have been optimized. The machine learning method itself can be a known method.

[0064] The learning model for region detection (region detection model) generated by the region detection learning processing unit 201 is preferably downloaded to the shooting terminal 3 when the application program for taking pictures of the display area is downloaded to the shooting terminal 3, as described later.

[0065] Furthermore, it is preferable for the region detection learning processing unit 201 to generate two learning models: a region detection model for regular sales areas and a region detection model for event sales areas. Therefore, it is preferable for the region detection learning data storage unit 200 to generate a region detection model (region detection model for regular sales areas) that has been machine-learned using image data of regular sales areas and annotation data tagging the product display areas and price tag display areas therein, and a region detection model (region detection model for event sales areas) that has been machine-learned using image data of event sales areas and annotation data tagging the event shelf areas 102 therein, and have these models downloaded to the shooting terminal 3.

[0066] The judgment model generation processing unit 21 generates a machine learning model to determine whether the image data captured by the shooting terminal 3 is suitable for analysis by the analysis processing unit 22 described later, that is, whether all or a predetermined area of ​​the image data captured by the shooting terminal 3 satisfies image quality conditions such as being free from blur and blur to a degree that can be analyzed by the analysis processing unit 22.

[0067] In other words, as will be described later, by inputting all of the image data of the regular sales area, or the image data of the product display area and / or price tag area, or all of the image data of the event sales area, or the image data of the event shelf area 102, into the learning model (network) generated by the judgment model generation processing unit 21, a machine learning model is generated to determine whether the captured image data, or the image data of the product display area and / or price tag area, or the image data of the event shelf area 102 in said image data, is free from blurring or blurring to an extent that can be analyzed by the analysis processing unit 22.

[0068] In the judgment model generation processing unit 21, it is preferable to use deep learning as the machine learning method. In this case, the learning model is a learning model in which the weighting coefficients between neurons in each layer of a neural network consisting of many hidden layers are optimized. Note that the machine learning used by the judgment model generation processing unit 21 is not limited to deep learning; other machine learning methods can also be used.

[0069] The judgment model generation processing unit 21 includes a judgment image data storage unit 210, a training image data generation processing unit 211, a judgment training data storage unit 212, and a judgment training processing unit 213.

[0070] The judgment image data storage unit 210 stores image data for generating machine learning image data in the learning image data generation processing unit 211, which will be described later. For example, it stores image data of various regular sales areas and event sales areas, image data extracted from image data of regular sales areas to extract product display areas and price tag areas, and image data extracted from image data of event sales areas to extract event shelf areas 102. The image data extracted from image data of regular sales areas to extract product display areas and price tag areas, and the image data extracted from image data of event sales areas to extract event shelf areas 102 may be actual images of each sales area to extract product display areas, price tag areas, and event shelf areas 102, or they may be images of those areas specifically identified as processing targets.

[0071] Furthermore, it is preferable that the image data stored in the judgment image data storage unit 210 is image data with sufficient image quality (sharp image quality) to be analyzeable by the analysis processing unit 22, without blurring or out of focus.

[0072] The learning image data generation processing unit 211 performs processing such as blurring and motion blurring on the image data stored in the judgment image data storage unit 210 using known methods. Blurring and motion blurring can be performed any number of times, but for example, it generates image data with motion blurring applied to sharp image data, image data with blurring applied in 1-pixel units to sharp image data, and image data with blurring applied in 3-pixel units to sharp image data. Known methods can be used for motion blurring and blurring.

[0073] This process yields four patterns of image data for the same subject: sharp image data (image data stored in the judgment image data storage unit 210), motion blur processed image data, 1-pixel blur processed image data, and 3-pixel blur processed image data. Furthermore, each image data is annotated with information indicating its image quality, such as Sharp (normal), Motion Blurred (blurred), Defocused (out of focus), and Slightly Defocused (slightly out of focus). Annotated data, consisting of the image data and annotations, is then generated as training image data. The generated training image data is stored in the judgment training data storage unit 212. The degree of blurring and out-of-focus areas should be determined by considering the degree to which the identification process (image recognition process) in the identification processing unit 222 of the analysis processing unit 22 (described later) is hindered, i.e., the degree to which misrecognition begins. "Defocused" means the image is slightly out of focus, making misrecognition highly likely. "Slightly Defocused" means the image is slightly out of focus, but misrecognition is still possible, though the possibility is low. "Motion Blurred" means the image is blurred, making misrecognition highly likely. "Sharp" means the image is almost completely free of misrecognition. These degrees can be distinguished using any number of levels and types.

[0074] The learning image data generation processing unit 211 generates motion-blurred image data and blurred image data from sharp image data. However, if out-of-focus or blurred image data is already available, it is sufficient to store it in the judgment learning data storage unit 212, and the judgment image data storage unit 210 and the learning image data generation processing unit 211 do not need to be provided.

[0075] Furthermore, while the learning image data generation processing unit 211 has been described above as performing blurring and motion blur processing to detect out-of-focus and blurry images, it may also perform other processing operations.

[0076] The judgment learning data storage unit 212 stores the learning image data generated by the learning image data generation processing unit 211. Specifically, it stores sharp image data of various regular sales areas, or image data extracted from image data of regular sales areas to extract the product display area and price tag area, or image data of event sales areas, or image data extracted from image data of event sales areas to extract the event shelf area 102, as well as image data that has been blurred and / or motion blurred, and annotation data that associates these with image quality annotations.

[0077] The judgment learning processing unit 213 inputs the learning data (annotation data) stored in the judgment learning data storage unit 212 as the correct answer to a learning model in which the weight coefficients between neurons in each layer of a neural network consisting of many hidden layers have been optimized, and performs the learning process. The machine learning method itself can be any known method.

[0078] The learning model for judgment (judgment model) generated by the judgment learning processing unit 213 is preferably downloaded to the shooting terminal 3 when the shooting terminal 3 downloads the application program for taking pictures of the display location, as described later.

[0079] Furthermore, it is preferable for the judgment learning processing unit 213 to generate judgment models for regular sales areas and judgment models for event sales areas as learning models. For this reason, the judgment image data storage unit 210 should store image data of regular sales areas or image data of regular sales areas from which the product display area and price tag area have been cut out, image data of event sales areas, or image data of event shelves 102 cut out from image data of event sales areas from which the event sales area has been photographed. In this case, the learning image data generation processing unit 211 generates image data by applying motion blur and blurring to image data of regular sales areas or image data of regular sales areas from which the product display area and price tag area have been cut out, and stores the image data with corresponding image quality annotations in the judgment learning data storage unit 212 as annotation data for regular sales areas. Furthermore, the learning image data generation processing unit 211 generates image data by applying motion blur and blurring to image data obtained by cutting out the event shelf area 102 from image data of the event sales area or image data of the event sales area, and stores it in the judgment learning data storage unit 212 as annotation data for the event sales area, along with image quality annotations.

[0080] The judgment learning data storage unit 212 stores annotation data that associates sharp image data for regular sales areas, image data that has been blurred and / or motion-blurred with that image data, and image quality annotations. The judgment image data storage unit 210 stores annotation data that associates sharp image data for event sales areas, image data that has been blurred and / or motion-blurred with that image data, and image quality annotations.

[0081] The judgment learning processing unit 213 generates a judgment learning model for regular sales areas by inputting annotation data for regular sales areas stored in the judgment learning data storage unit 212 as the correct answer and performing machine learning, and then downloads it to the camera terminal 3. In addition, the judgment learning processing unit 213 generates a judgment learning model for event sales areas by inputting annotation data for event sales areas stored in the judgment learning data storage unit 212 as the correct answer and performing machine learning, and then downloads it to the camera terminal 3.

[0082] The analysis processing unit 22 receives image data (captured image data) taken by the camera terminal 3 of the regular sales area and the event sales area, and performs the process of identifying the products and / or prices shown in the received captured image data.

[0083] The analysis processing unit 22 includes a captured image data receiving processing unit 220, a captured image data storage unit 221, and an identification processing unit 222.

[0084] The image data receiving processing unit 220 receives image data taken by the shooting terminal 3 of the regular sales area and the event sales area, and stores it in the image data storage unit 221. As will be described later, the received image data is determined by the judgment processing unit 34 to be free from blur and blur to the extent that it can be analyzed by the analysis processing unit 22, so the image quality is such that errors are less likely to occur during the analysis processing by the analysis processing unit 22.

[0085] The captured image data storage unit 221 stores the captured image data received by the captured image data receiving processing unit 220.

[0086] The identification processing unit 222 identifies the products in the product display area or event shelf area 102, the price tag area or the price of the products in the event shelf area 102, etc., of the captured image data stored in the captured image data storage unit 221. The identification processing unit 222 can use known methods. For the identification process of products, prices, etc., image matching and machine learning processing can be used.

[0087] In the case of machine learning processing, a learning model in which the weight coefficients between neurons in each layer of a neural network consisting of many hidden layers are optimized is trained by inputting the products shown in the product display area, the prices on the price tags in the price tag area, and the products shown in the event shelf area 102 as correct answers. Then, by inputting the captured image data from the captured image data storage unit 221 to this learning model, the products shown in the product display area and / or the prices on the price tags in the price tag area, the products shown in the event shelf area 102 and / or the prices on the price tags, etc. The machine learning method itself can be a known method.

[0088] The camera terminal 3 is a portable communication terminal operated by a photographer to photograph the regular sales areas and event sales areas of a store, as well as the displayed products and their price tags. The portable communication terminal has application software installed to realize the processing in the camera terminal 3 of the information processing system 1 of the present invention. The photographer activates and uses this application software to realize the processing in the camera terminal 3 of the present invention.

[0089] The shooting terminal 3 includes an shooting processing unit 30, a region detection model storage unit 31, a region detection processing unit 32, a determination model storage unit 33, a determination processing unit 34, and a transmission processing unit 35.

[0090] The shooting processing unit 30 activates a shooting device such as a camera installed in the shooting terminal 3 and uses the shooting device to photograph the regular sales area and the event sales area. Furthermore, when the area detection processing unit 32, which will be described later, has identified the area to be photographed, it acquires image data of the regular sales area and the event sales area by pressing the shutter of the shooting device or extracting still image information from the moving image.

[0091] The region detection model storage unit 31 stores the region detection model (machine learning model) generated by the region detection model generation processing unit 20 of the management terminal 2. The region detection model storage unit 31 may store region detection models for regular sales areas and region detection models for event sales areas, or it may store only one of them.

[0092] The area detection processing unit 32 detects the product display area and the price tag area from the image data captured by the shooting processing unit 30 using the shooting device, which captures the regular sales area. In addition, the shooting processing unit 30 detects the event shelf area 102 from the image data captured by the shooting device, which captures the event sales area.

[0093] When the shooting processing unit 30 receives a shooting start operation from the photographer and acquires image data captured by the shooting device, the region detection processing unit 32 takes that image data (for example, each image data of 20 frames per second, or a portion of the image data thereafter) as input values ​​and inputs them into the region detection model stored in the region detection model storage unit 31, thereby detecting the product display area, price tag area, and event shelf area 102. It is preferable that the region detection processing unit 32 be configured to allow the extraction of the product display area, price tag area, and event shelf area 102 detected from the image data captured by the shooting processing unit 30.

[0094] The judgment model storage unit 33 stores the judgment model (machine learning model) generated by the judgment model generation processing unit 21 of the management terminal 2. The judgment model storage unit 33 may store judgment models for regular sales areas and judgment models for event sales areas, or it may store only one of them.

[0095] The judgment processing unit 34 uses the judgment model stored in the judgment model storage unit 33 to determine whether the image data captured by the shooting processing unit 30 is out of focus, blurred, etc. Figure 4 shows an overview of the processing in the judgment processing unit 34.

[0096] The judgment processing unit 34 sets one or more verification areas 101 of a predetermined size in the image data captured by the shooting processing unit 30, and inputs the image data in the verification areas 101 as input values ​​to the judgment model in the judgment model storage unit 33, thereby determining the image quality of the verification areas 101. For example, it determines the image quality of the image data in the verification areas 101 as Sharp, Motion Blurred, Defocused, Slightly Defocused, etc.

[0097] The size of the verification area 101 can be set arbitrarily. For example, when photographing so that the entire shelf of a display rack fits within the field of view of the camera, and assuming the image resolution of the camera is approximately 3000 pixels wide x 4000 pixels high, the size of the verification area 101 can be set to 240 pixels high and 240 pixels wide when photographing vertically, and 320 pixels high and 320 pixels wide when photographing horizontally, as an example.

[0098] The determination processing unit 34 determines whether the image quality of the image data in each verification area 101 satisfies predetermined conditions, and displays a message indicating whether the image was captured successfully, whether to prompt for reshooting, or whether to leave the decision of reshooting to the photographer.

[0099] For example, if the majority of the verification areas 101 in the image data are judged to have sharp image quality, then it is determined that the image was captured successfully.

[0100] For example, if the majority of the verification areas 101 in the image data are determined to have Motion Blurred or Defocused image quality, the determination processing unit 34 displays a message prompting reshooting on the display device 72 of the shooting terminal 3.

[0101] For example, if the majority of the verification areas 101 in the image data are determined to have slightly defocused image quality, the determination processing unit 34 displays a message on the display device 72 of the shooting terminal 3 indicating that the decision of whether or not to retake the image is left to the photographer.

[0102] As shown in Figure 4, when photographing display shelves in a regular sales area, for example, a predetermined size of verification area 101 is set near the upper left and right edges of the product display area or price tag area on the top shelf, near the lower left and right edges of the product display area or price tag area on the bottom shelf, and near the center of the image data, and the image quality of each verification area 101 is determined. Figure 4(a) is an example of image data of a display shelf in a regular sales area taken by the shooting processing unit 30, and Figure 4(b) shows the verification area 101.

[0103] Furthermore, as shown in Figure 5, when photographing display shelves in a special event sales area, for example, a predetermined area is searched for and set as the verification area 101 within the special event shelf area 102 of the special event sales area, in the direction furthest from the center of the rectangular area 103 containing the special event shelf area 102 in the direction of the upper left and lower left and right edges. For example, the search for the verification area 101 is performed by shifting a certain distance from the center of the rectangular area 103 in the direction of the upper left and lower left and right edges of the verification area 101, and when the furthest point within the inclusion relationship is reached, that point is set as the verification area 101. Figure 5(a) is an example of image data of display shelves in a special sale sales area taken by the shooting processing unit 30, Figure 5(b) is a diagram showing the search process for the verification area 101, and Figure 6 is a diagram showing the verification area 101.

[0104] Another method involves assuming 45-degree diagonal lines 104 touching the left, right, top, and bottom edges of the detection area, starting the search for the verification area 101 from the position touching each of these diagonal lines 104, and gradually searching the verification area 101 toward the center of the rectangular area 103 that includes the detection area. This is schematically shown in Figure 7.

[0105] If the determination processing unit 34 determines that the image was captured successfully, or if the photographer has instructed that reshooting is not necessary, the transmission processing unit 35 stores the image data captured by the shooting processing unit 30, or the image data of the product display area and / or shelf area in said image data, in a predetermined storage area. Then, at a predetermined timing, it sends the stored image data to the management terminal 2.

[0106] The image data transmitted by the transmission processing unit 35 has been determined by the judgment processing unit 34 to be free from blur and blur to the extent that it can be analyzed by the analysis processing unit 22. Therefore, the image quality is such that errors are less likely to occur during the analysis processing by the analysis processing unit 22. As a result, the possibility of having to retake the image is reduced. [Examples]

[0107] Next, an example of processing using the information processing system 1 of the present invention will be explained using the flowchart in Figure 3.

[0108] First, I will explain the process of generating the region detection model.

[0109] The learning data storage unit 200 for area detection stores image data of various regular sales areas and annotation data in which the product display area and price tag area are tagged in that image data, as well as image data of various event sales areas and annotation data in which the event shelf area 102 is tagged in that image data.

[0110] The region detection learning processing unit 201 receives an operation from the management terminal 2 to generate a region detection model for the regular sales area. It then extracts image data of the regular sales area and annotation data tagging product display areas and price tag display areas from that image data, which are stored in the region detection learning data storage unit 200, and uses this to generate a learning model that performs region detection on the image data of the regular sales area.

[0111] Furthermore, the region detection learning processing unit 201 receives an operation from the management terminal 2 to generate a region detection model for the event sales area. It then extracts image data of the event sales area and annotation data tagging the event shelf area 102 in that image data, which are stored in the region detection learning data storage unit 200, and uses this to generate a learning model that performs region detection on the image data of the event sales area.

[0112] The learning models that perform area detection for the generated regular sales areas and the learning models that perform area detection for the event sales areas are stored in a designated memory area.

[0113] Next, we will explain the process of generating the decision model.

[0114] The judgment image data storage unit 210 stores image data of various regular sales areas, or image data extracted from the image data of regular sales areas, specifically the product display area and price tag area. It also stores image data of various event sales areas, or image data extracted from the image data of event sales areas, specifically the event shelf area 102.

[0115] When the learning image data generation processing unit 211 receives a request for the generation of learning image data from the management terminal 2, it extracts image data of various regular sales areas, or image data extracted from the image data of regular sales areas, specifically the product display area and price tag area, from the judgment image data storage unit 210. It also extracts image data of various event sales areas, or image data extracted from the image data of event sales areas, specifically the event shelf area 102.

[0116] Then, each extracted image data is subjected to image quality processing such as blurring and motion blur. In addition, the image data that has undergone image quality processing is annotated with information that indicates the image quality corresponding to the processing. For example, if motion blur processing is performed, "Motion Blurred" is annotated; if 1 pixel blur is performed, "Slightly Defocused" is annotated; and if 3 pixels blur is performed, "Defocused" is annotated.

[0117] Regarding image processing, any processing can be applied, and information corresponding to that processing should be annotated.

[0118] Furthermore, the learning image data generation processing unit 211 annotates each extracted image data that has not undergone image quality processing, with information such as "Sharp" indicating that the image quality is free from blur, blur, etc.

[0119] Then, the annotation data, which consists of image data and annotations, is stored in the judgment learning data storage unit 212.

[0120] The judgment learning processing unit 213 receives an operation from the management terminal 2 to generate a judgment model for regular sales areas. It then extracts image data of regular sales areas, or image data extracted from the image data of regular sales areas, along with annotation data that includes information indicating the image quality, from the judgment learning data storage unit 212. Using this data, it generates a learning model for performing judgment processing on the image data of regular sales areas.

[0121] Furthermore, the judgment learning processing unit 213 receives an operation from the management terminal 2 to generate a judgment model for the event sales area. It then extracts image data of the event sales area, or image data extracted from the image data of the event sales area, along with annotation data indicating the image quality, from the judgment learning data storage unit 212, and uses this to generate a learning model for performing judgment processing on the image data of the event sales area.

[0122] The learning models that perform the determination of whether a sales area is a regular sales area and the learning models that perform the determination of whether a sales area is a special event sales area are stored in a designated memory area.

[0123] A photographer who takes pictures of regular sales areas, event sales areas, etc. in a store installs a predetermined application program on the shooting terminal 3 used in the information processing system 1 of the present invention and makes it ready for use before commencing the shooting work.

[0124] This application program includes a learning model for detecting the area of ​​a regular sales area, a learning model for detecting the area of ​​a special event sales area, both generated by the area detection model generation processing unit 20, and a learning model for performing judgment processing on a regular sales area, both generated by the judgment model generation processing unit 21. Upon installation, these learning models become available for use on the camera terminal 3.

[0125] When a photographer takes pictures of regular sales areas, event sales areas, etc., in a store using the photography terminal 3 of the information processing system 1 of the present invention, they launch the application program installed on the photography terminal 3. Then, they select whether to photograph the regular sales area or the event sales area.

[0126] This instruction is received by the application program on the camera terminal 3. If it is confirmed that the photo was taken in a regular sales area, the area detection processing unit 32 and the judgment processing unit 34 will input the information to the learning model for regular sales areas. If it is confirmed that the photo was taken in a special event sales area, the area detection processing unit 32 and the judgment processing unit 34 will input the information to the learning model for special event sales areas.

[0127] First, we will explain the processing performed by the camera terminal 3 when it receives instructions to photograph the regular sales area.

[0128] The shooting processing unit 30 receives an instruction from the photographer to take a photograph of the regular sales area and activates the shooting device on the shooting terminal 3 (S100). When the photographer points the shooting device on the shooting terminal 3 towards the regular sales area, the shooting device takes a photograph of the regular sales area (S110), and the shooting processing unit 30 receives the image data (for example, each image data of 20 frames per second, or a portion of the image data). The shooting processing unit 30 displays the received image data on the display device 72 of the shooting terminal 3. An example of the screen at this time is shown in Figure 8. In Figure 8, a display shelf, which is a regular sales area, is photographed, and guidelines 105 are displayed to make it easier to take a photograph in accordance with the width of the display shelf so that the photograph can be taken appropriately.

[0129] Therefore, photographers taking pictures of regular sales areas and those taking pictures of display shelves should preferably take the pictures from a position where the left and right edges of the display shelves are aligned with guideline 105.

[0130] The area detection processing unit 32 then uses the image data as input to the area detection model for standard sales areas stored in the area detection model storage unit 31, thereby detecting the product display area and price tag area (S120). Figure 9 shows an example of a screen displaying the detected areas. For clarity, the product display area and price tag area are highlighted in the screen shown in Figure 9.

[0131] As described above, the region detection processing unit 32 takes the image data of each frame received from the shooting device by the shooting processing unit 30 as input values ​​and inputs it into the region detection model for regular sales areas stored in the region detection model storage unit 31, thereby identifying and displaying the product display area and price tag area in the image data.

[0132] Furthermore, while Figures 8 and 9 show the case where the camera terminal 3 is held vertically to photograph the entire display shelf in the regular sales area, the camera terminal 3 may also be held horizontally to take photographs, as shown in Figure 10. In this case, for example, it is possible to enlarge and photograph a part of the shelf in the regular sales area. Note that, as with Figure 9, the product display area and price tag area are highlighted in the screen of Figure 10 for clarity.

[0133] Then, when the photographer performs an image data recording operation (shooting operation), such as pressing the shutter button on the shooting terminal 3 (S130), the shooting processing unit 30 records the image data in a predetermined storage area (S140). The area detection processing unit 32 also identifies the product display area and price tag area detected in the recorded image data.

[0134] Then, regarding the image data recorded in S140, the judgment processing unit 34 performs a judgment process to determine whether there is out-of-focus, blur, etc., using the judgment model for regular sales areas stored in the judgment model storage unit 33 (S150).

[0135] For example, the judgment processing unit 34 sets predetermined verification areas 101 in the image data recorded in S140, such as near the left and right edges of the uppermost product display area or price tag area, near the left and right edges of the lowermost product display area or price tag area, and near the center of the image data, and inputs the image data of each verification area 101 as input values ​​to the judgment model for regular sales areas stored in the judgment model storage unit 33. This is schematically shown in Figure 11. In Figure 11, verification areas 101a and 101b are set near the left and right edges of the uppermost product display area, verification areas 101d and 101f are set near the left and right edges of the lowermost price tag area, and verification area 101c is set near the center of the image data, all with predetermined sizes, and the image data of each of these verification areas 101a to 101e is input to the judgment model for regular sales areas to determine whether the image quality satisfies predetermined conditions (S160).

[0136] For example, in this determination, for each verification area 101, a result of image quality determination such as Sharp, Motion Blurred, Defocused, or Slightly Defocused is obtained. If the majority of the verification areas 101 in the image data are determined to have Sharp image quality (S160), it is determined that the image was captured successfully, and the image data received in S140 or a part of it (for example, image data of the product display area and price tag area) is stored in a predetermined storage area (S170). An example of the screen in this state is shown in Figure 12.

[0137] If the majority of the verification areas 101 in the image data are determined to have Motion Blurred or Defocused image quality (S160), the determination processing unit 34 decides to reshoot and displays a message prompting reshoot on the display device 72 of the shooting terminal 3 (S180). An example of the screen in this state is shown in Figure 13. Then, the image data received in S140 is discarded, and the processing from S110 onwards is repeated.

[0138] If the majority of the verification areas 101 in the image data are determined to have slightly defocused image quality (S160), the determination processing unit 34 decides to leave the decision of whether or not to retake the image to the photographer and displays a message prompting the photographer to decide whether or not to retake the image on the display device 72 of the shooting terminal 3 (S190). An example of the screen in this state is shown in Figure 14. If the photographer chooses to store the image data as is (do not retake the image) (S190), the image data received in S140 or a part of it (for example, image data of the product display area and price tag area) is stored in a predetermined storage area. On the other hand, if the photographer chooses to retake the image, the image data received in S140 is discarded, and the processing from S110 onwards is repeated.

[0139] The transmission processing unit 35 then sends the image data stored in the memory area or a part of it (for example, image data of the product display area or price tag area) to the management terminal 2 at a predetermined timing (S200). At this time, information necessary for the analysis process described later, such as the date and time of shooting, store identification information such as the store name, and information indicating the location of the sales area where the image was taken, may also be transmitted in association with the image data.

[0140] Next, we will explain the processing performed by the camera terminal 3 when it receives instructions to take photographs of the event sales area.

[0141] The shooting processing unit 30 receives an instruction from the photographer to take a photograph of the event sales area and activates the shooting device on the shooting terminal 3 (S100). When the photographer points the shooting device on the shooting terminal 3 towards the event sales area, the shooting device takes a photograph of the event sales area (S110), and the shooting processing unit 30 receives the image data (for example, each image data of 20 frames per second, or a portion of the image data). The shooting processing unit 30 displays the received image data on the display device 72 of the shooting terminal 3. An example of the screen at this time is shown in Figure 15. In Figure 15, the display shelves of the event sales area are photographed. Unlike regular sales areas, event sales areas generally have an irregular shape, so it is preferable not to display the guideline 105 as in regular sales areas, however, guideline 105 or areas to guide the position may be displayed.

[0142] Then, the region detection processing unit 32 detects the event shelf region 102 by inputting the image data as an input value into the region detection model for the event sales area stored in the region detection model storage unit 31 (S120). Figure 5(a) shows an example of a screen displaying the detected region.

[0143] As described above, the region detection processing unit 32 takes the image data of each frame received from the shooting device by the shooting processing unit 30 as input values ​​and inputs it into the region detection model for the event sales area stored in the region detection model storage unit 31, thereby identifying and displaying the event shelf region 102 in the image data.

[0144] Then, when the photographer performs an image data recording operation (shooting operation), such as pressing the shutter button on the shooting terminal 3 (S130), the shooting processing unit 30 records the image data in a predetermined storage area (S140). The area detection processing unit 32 also identifies the event shelf area 102 that has been detected in the recorded image data.

[0145] Then, regarding the image data recorded in S140, the judgment processing unit 34 performs a judgment process to determine whether there is out-of-focus, blur, etc., using the judgment model for the event sales area stored in the judgment model storage unit 33 (S150).

[0146] For example, the determination processing unit 34 searches for and sets a predetermined verification area 101 in the image data recorded in S140, for example, a predetermined area in the event shelf area 102 of an event sales area that is furthest from the center of the rectangular area 103 containing the event shelf area 102 in the direction of the upper left and right edges and the lower left and right edges as the verification area 101.

[0147] The image data from each set verification area 101 is then input as input values ​​to the judgment model for the event sales area, which is stored in the judgment model storage unit 33. This is schematically shown in Figure 16. In Figure 16, verification areas 101a and 101b are set near the upper left and right ends of the event shelf area 102, verification areas 101d and 101f are set near the lower left and right ends of the event shelf area 102, and verification area 101c is set near the center of the image data, all with predetermined sizes. The image data from each of these verification areas 101a to 101e is input to the judgment model for the event sales area to determine whether the image quality satisfies predetermined conditions (S160).

[0148] For example, in this determination, for each verification area 101, a result of image quality determination such as Sharp, Motion Blurred, Defocused, or Slightly Defocused is obtained. If the majority of the verification areas 101 in the image data are determined to have Sharp image quality (S160), it is determined that the image was captured successfully, and the image data received in S140 or a part of it (for example, the image data of the event shelf area 102) is stored in a predetermined storage area (S170). An example of the screen in this state is shown in Figure 12.

[0149] If the majority of the verification areas 101 in the image data are determined to have Motion Blurred or Defocused image quality (S160), the determination processing unit 34 decides to reshoot and displays a message prompting reshoot on the display device 72 of the shooting terminal 3 (S180). An example of the screen in this state is shown in Figure 13. Then, the image data received in S140 is discarded, and the processing from S110 onwards is repeated.

[0150] If the majority of the verification areas 101 in the image data are determined to have slightly defocused image quality (S160), the determination processing unit 34 decides to leave the decision of whether or not to retake the image to the photographer and displays a message prompting the photographer to decide whether or not to retake the image on the display device 72 of the shooting terminal 3 (S190). An example of the screen in this state is shown in Figure 14. If the photographer chooses to store the image data as is (do not retake the image) (S190), the image data received in S140 or a part of it (for example, the image data of the event shelf area 102) is stored in a predetermined storage area. On the other hand, if the photographer chooses to retake the image, the image data received in S140 is discarded, and the processing from S110 onwards is repeated.

[0151] The transmission processing unit 35 then sends the image data stored in the memory area or a part of it (for example, image data of the event shelf area 102) to the management terminal 2 at a predetermined timing (S200). At this time, information necessary for the analysis process described later, such as the date and time of shooting, store identification information such as the store name, and information indicating the location of the sales area where the image was taken, may also be transmitted in association with the image data.

[0152] If the determination processing unit 34 determines that the image was captured successfully, or if it receives instructions from the photographer that reshooting is not necessary, the shooting processing unit 30 stores the captured image data, or the image data of the product display area, shelf area, and / or event shelf area 102 in the captured image data, in a predetermined storage area. The transmission processing unit 35 then sends the image data stored in the predetermined storage area to the management terminal 2 at a predetermined timing.

[0153] The image data receiving processing unit 220 of the management terminal 2 receives image data of the regular sales area and / or image data of the product display area or price tag area in the image data of the regular sales area, image data of the event sales area and / or image data of the event shelf area 102 in the image data of the event sales area, sent by the shooting terminal 3 in S200, and stores it in the image data storage unit 221.

[0154] Then, at a predetermined timing, the identification processing unit 222 extracts image data of the regular sales area and / or image data of the product display area or price tag area in the image data of the regular sales area stored in the captured image data storage unit 221, image data of the event sales area and / or image data of the event shelf area 102 in the image data of the event sales area, and performs analysis processing to identify products, prices, etc. in the extracted image data (S210).

[0155] For example, the extracted image data is input to a learning model for analysis processing to identify products in the product display area or event shelf area 102, the prices on the price tags in the price tag area or event shelf area 102, etc.

[0156] By performing the above processing, the analysis processing unit 22 uses image data that has been determined to be free from blur and other issues to perform product and price analysis, thus reducing the likelihood of errors in the analysis processing unit 22. Furthermore, since the system assists in capturing image data that will be properly analyzed, it reduces the burden of having to re-shoot images if an error is discovered during the analysis process. [Examples]

[0157] In the above-described embodiment 1, a case was explained in which five regions were used as the verification region 101 in the judgment processing unit 34, but there may be more or fewer regions. Reducing the number of verification regions 101 can reduce the processing time. To reduce processing time and maintain verification accuracy, it is preferable to have about three regions. For example, verification regions 101 may be provided near the upper left and right ends and near the right or left end of the lower end, or near the lower left and right ends and near the right or left end of the upper end.

[0158] It is preferable to position each verification region 101 near the edge, and as far away as possible from the center of the image data. This is because the focus tends to be near the center, and often shifts near the edges.

[0159] Furthermore, as shown in Figure 10, when the shooting terminal 3 is held horizontally and shooting is performed, that is, when shooting at close range, only about one or two areas of the product display area and price tag area are often captured. Therefore, it is preferable to capture the product display area and / or price tag area near the center and near the upper left and right edges, and near the center and near the lower left and right edges.

[0160] Regarding the determination process in the determination processing unit 34, in the above-described embodiment 1, the determination results of image quality in the majority of the verification areas 101 were used, but the determination result of the most frequent image quality may also be used. Furthermore, these determination results may be weighted, and the determination results of the verification areas 101 near the edges may be given more weight than those of the verification areas 101 near the center. [Examples]

[0161] As a variation of the above-described embodiments, the information processing system 1 in this embodiment may have a shooting condition processing unit 36 ​​that determines whether the image data satisfies predetermined shooting conditions in conjunction with the region detection processing of the region detection processing unit 32 at the shooting terminal 3. An example of the overall processing functions of the information processing system 1 in this embodiment is shown in a block diagram in Figure 17. An example of the overall processing of the information processing system 1 in this embodiment is shown in Figure 18.

[0162] The shooting condition processing unit 36 ​​determines whether the shooting conditions for correctly capturing the subject are met. Specifically, it determines whether the entire display shelf to be captured is contained within the image data at a sufficiently large size without being cut off, whether the shooting terminal 3 is facing upwards or downwards, causing the upper or lower edges of the subject to be captured to appear small in the image (downward shooting, upward shooting), and whether the shooting position is shifted laterally relative to the shelf, causing the shelf to appear distorted (diagonal shooting).

[0163] To determine whether the entire display shelf to be photographed is contained within the image data at a sufficiently large size without extending beyond the frame, the shooting condition processing unit 36 ​​detects the edges of the display shelf from the image data and determines whether the edges are within the range of the guideline 105 in the image data, or whether the width direction of the product display area and price tag area detected by the area detection processing unit 32 is within the range of the guideline 105. Then, by determining whether the width direction of the product display area and price tag area detected by the area detection processing unit 32 is at a certain ratio or greater to the width direction length of the guideline 105, it determines whether the image is captured at a predetermined size or larger. This is schematically shown in Figure 19. In Figure 19, the shaded area indicates that it is not contained within the range of the guideline 105.

[0164] Furthermore, to determine whether the upper and lower edges of the subject being photographed are too small due to the camera terminal 3 being pointed upwards or downwards, the gyro sensor in the camera terminal 3 is used to determine whether the shooting direction is upwards or downwards. Alternatively, the region detection processing unit 32 may determine, for example, whether the positions of the left and right edges of the product display area and price tag area detected by the region detection processing unit 32 narrow downwards (downward shooting) or upwards (upward shooting). The deviation of the shooting position relative to the display shelf can also be determined by whether the left and right edges of the product display area and price tag area detected by the region detection processing unit 32 are shifted inward downwards. Figure 20 shows an example of a warning displayed when "diagonal shooting" is performed.

[0165] The shooting condition processing unit 36 ​​should display an alert if any of the above conditions are not met.

[0166] Furthermore, until the shooting condition processing unit 36 ​​determines that the conditions are met, the shooting processing unit 30 should perform control processing to prevent shooting operations (image data recording operations), such as pressing the shooting button. Then, as shown in Figure 21, once the shooting condition processing unit 36 ​​determines that all conditions are met, the shooting processing unit 30 should perform control processing to enable shooting operations, such as pressing the shooting button.

[0167] Next, an example of the processing of the information processing system 1 in this embodiment will be explained using the flowchart in Figure 18.

[0168] Note that the process for generating the region detection model and the process for generating the judgment model are the same as in Example 1, so we will omit the explanation.

[0169] Similar to Example 1, the application program is installed on the shooting terminal 3. The learning model for region detection and the learning model for judgment processing are also available for use on the shooting terminal 3.

[0170] When a photographer takes pictures of regular sales areas, event sales areas, etc., in a store using the photography terminal 3 of the information processing system 1 of the present invention, they launch the application program installed on the photography terminal 3. Then, they select whether to photograph the regular sales area or the event sales area.

[0171] This instruction is received by the application program on the camera terminal 3. If it is confirmed that the photo was taken in a regular sales area, the area detection processing unit 32 and the judgment processing unit 34 will input the information to the learning model for regular sales areas. If it is confirmed that the photo was taken in a special event sales area, the area detection processing unit 32 and the judgment processing unit 34 will input the information to the learning model for special event sales areas.

[0172] First, we will explain the processing performed by the camera terminal 3 when it receives instructions to photograph the regular sales area.

[0173] The shooting processing unit 30 receives an instruction from the photographer to take a photograph of the regular sales area and activates the shooting device on the shooting terminal 3 (S300). When the photographer points the shooting device on the shooting terminal 3 towards the regular sales area, the shooting device takes a photograph of the regular sales area (S310), and the shooting processing unit 30 receives the image data (for example, each image data of 20 frames per second, or a portion of the image data thereafter). The shooting processing unit 30 displays the received image data on the display device 72 of the shooting terminal 3.

[0174] Therefore, photographers taking pictures of regular sales areas and those taking pictures of display shelves should preferably take the pictures from a position where the left and right edges of the display shelves are aligned with guideline 105.

[0175] Then, the area detection processing unit 32 takes the image data as input values ​​and inputs them into the area detection model for regular sales areas stored in the area detection model storage unit 31 to detect the product display area and the price tag area (S320).

[0176] As described above, the region detection processing unit 32 takes the image data of each frame received from the shooting device by the shooting processing unit 30 as input values ​​and inputs it into the region detection model for regular sales areas stored in the region detection model storage unit 31, thereby identifying and displaying the product display area and price tag area in the image data.

[0177] Then, the shooting condition processing unit 36 ​​determines whether the image data being captured satisfies predetermined conditions (S330).

[0178] The shooting condition processing unit 36 ​​detects the gap in the display shelf from the image data and determines whether the gap is within the range of the guideline 105 in the image data, or whether the width direction of the product display area and price tag area detected by the area detection processing unit 32 is within the range of the guideline 105. Then, by determining whether the width direction of the product display area and price tag area detected by the area detection processing unit 32 is at a certain ratio or greater to the width direction length of the guideline 105, it determines whether the image is captured at a predetermined size or larger.

[0179] If these conditions are not met, for example, as shown in Figure 19, the camera will notify the photographer of areas that have not been properly photographed by shading the product display area or price tag area that does not meet the conditions with a predetermined color (S340). It is also advisable to change the display of areas that have not been properly photographed, such as product display areas or price tag areas that do not meet the conditions, by changing the display of the areas that have not been properly photographed, such as shading, highlighting, or enclosing them in a frame, so that the photographer can see the areas that have not been properly photographed.

[0180] Furthermore, the shooting condition processing unit 36 ​​determines whether the shooting orientation of the object to be photographed is correct. For example, it uses sensors such as the gyro sensor in the shooting terminal 3 to determine whether the shooting orientation is upward or downward, and also determines whether the shooting orientation in the vertical direction is correct by determining, for example, whether the left and right edges of the product display area and price tag area detected by the area detection processing unit 32 narrow as they go downward (downward shooting) or as they go upward (upward shooting). In addition, it determines whether it is diagonal shooting by determining whether the left and right edges of the product display area and price tag area detected by the area detection processing unit 32 are moving inward as they go downward. A warning for diagonal shooting is shown in Figure 20.

[0181] As described above, the shooting condition processing unit 36 ​​determines whether the conditions for shooting the image to be captured by the shooting processing unit 30 are met. If the conditions are not met, a warning is displayed; if the conditions are met, shooting is permitted.

[0182] Then, when the photographer performs an image data recording operation (shooting operation), such as pressing the shutter button on the shooting terminal 3 (S350), the shooting processing unit 30 records the image data in a predetermined storage area (S360). The area detection processing unit 32 also identifies the product display area and price tag area detected in the recorded image data.

[0183] The subsequent processing is the same as in Example 1, so the explanation will be omitted.

[0184] Furthermore, the shooting condition determination process and warning display process performed by the shooting condition processing unit 36 ​​can be applied not only when photographing regular sales areas but also when photographing event sales areas.

[0185] For example, by determining whether the width of the event area is within the range of guideline 105 and is at a certain ratio or greater to the length of guideline 105 in the width direction, it is possible to determine whether the image data does not extend beyond the designated area and is captured at a size greater than or equal to the specified size.

[0186] Furthermore, sensors such as a gyro sensor can be used to detect whether the shooting device 3 is facing upwards or downwards, and to determine whether the shot is upwards or downwards. [Examples]

[0187] In each of the embodiments described above, the order of processing can be changed as appropriate, and some processing can be omitted or added. For example, the image processing unit 30 may perform a correction process to correct the image data so that it is taken from a position directly facing the image data it has captured. If this correction process is performed at the image capture terminal 3, it can be performed at any time before the transmission processing unit 35 corrects the image data. Alternatively, the correction process may be performed before the identification process of the identification processing unit 222 in the analysis processing unit 22 of the management terminal 2.

[0188] In the embodiments described above, the case in which the region detection model and judgment model are operated on the shooting terminal 3 was explained, but the management terminal 2 may also have a region detection model storage unit 31 and a region detection processing unit 32, a judgment model storage unit 33 and a judgment processing unit 34. In this case, the image data captured by the shooting processing unit 30 is sent to the management terminal 2, the management terminal 2 performs region detection processing and judgment processing, and the results are returned to the shooting terminal 3. Alternatively, the management terminal 2 may be equipped with either the region detection model for regular sales areas or the region detection model for event sales areas, or either the judgment model for regular sales areas or the judgment model for event sales areas, and some processing may be performed on the management terminal 2.

[0189] In the embodiments described above, the image quality determination process used four levels of image quality, such as Sharp, Motion Blurred, Defocused, and Slightly Defocused. However, the number of image quality levels may be fewer or more than four.

[0190] In the judgment processing unit 34, we have described the case where the verification area 101 is set near the left, right, top, bottom edges and near the center. However, if a specific object or a specific area is detected, that area can also be set as the verification area 101.

[0191] For example, in the case of image data for a regular sales area, in addition to the product display area and / or price tag area, individual price tags are trained to be detected, and the area detection model of the area detection processing unit 32 is set to detect the product display area and / or price tag area, as well as the area of ​​individual price tags. Part or all of the area of ​​each individual price tag may be set as the verification area 101.

[0192] Furthermore, the photographer may input and accept the setting for the area to be designated as verification area 101.

[0193] Furthermore, it is preferable that the entire verification area 101 is included in the product display area, price tag area, and event area, but it is also acceptable for only a part of the verification area 101 to be included in the product display area, price tag area, and event area.

[0194] While the explanation above describes a shooting terminal 3 as a device used by the photographer to take pictures, other cameras, such as security cameras in stores, can also be used as shooting terminal 3. [Industrial applicability]

[0195] By using the information processing system 1 of the present invention, it becomes possible to assist photographers in capturing images that are free from blur or smudges that could hinder image recognition processing, such as when photographing display shelves or other surfaces where products are displayed. [Explanation of symbols]

[0196] 1: Information Processing System 2: Management terminal 3: Shooting device 20: Region detection model generation processing unit 21: Decision Model Generation Processing Unit 22: Analysis Processing Unit 30: Shooting Processing Unit 31: Region detection model storage unit 32: Region detection processing unit 33: Decision Model Memory Unit 34: Determination Processing Unit 35: Transmission Processing Unit 36: Shooting Condition Processing Section 70: Arithmetic device 71:Storage device 72:Display device 73: Input device 74: Communication equipment 101: Verification Area 102: Event display shelf area 103: Rectangular area including event display shelf area 104: Line segment tangent to the event display shelf area 105: Guidelines 200: Storage unit for learning data used for region detection 201: Region detection learning processing unit 210: Image data storage unit for judgment 211: Learning Image Data Generation Processing Unit 212: Storage unit for learning data used for judgment 213: Judgment Learning Processing Unit 220: Image Data Reception Processing Unit 221: Image data storage unit 222: Identification Processing Unit

Claims

1. An information processing system that assists in photographing product displays, The aforementioned information processing system is A shooting processing unit that photographs the sales area with the shooting device of the shooting terminal, A region detection processing unit detects a predetermined region by inputting the image data captured by the aforementioned image capture processing unit into a learning model that detects a predetermined region from the image data captured by the aforementioned image capture processing unit, A determination model for determining whether image data satisfies image quality conditions to an extent that allows for analysis is input image data of a verification region to be targeted for image quality determination processing, which is set in part or all of a predetermined region detected by the region detection processing unit, and a determination processing unit that performs image quality determination processing, An information processing system characterized by having the following features.

2. The determination processing unit, If the image quality of the verification area satisfies the predetermined conditions, the image data is stored as the image data to be processed. The information processing system according to feature 1.

3. The determination processing unit, If the image quality of the verification area does not satisfy the predetermined conditions, the system prompts or has the system perform a re-shoot of the image data. The information processing system according to feature 2.

4. The determination processing unit, If the image quality of the verification area does not satisfy the predetermined conditions, a message is displayed prompting the user to choose whether to store the image data as the image data to be processed or to re-capture it, and the user accepts the instruction. The information processing system according to feature 3.

5. The determination processing unit, Multiple verification areas are set, The image quality is determined for each of the aforementioned multiple verification areas. The processing of the image data is changed according to the number of image quality levels determined for each verification area. The information processing system according to feature 2.

6. The determination processing unit, For each of the multiple verification areas set above, one or more of the following image quality conditions are determined: normal, blurry, out of focus, slightly out of focus. If the number of verification regions in which the image quality is determined to be normal is more than half or a large number, the image data is stored as the image data to be processed. The information processing system according to feature 5.

7. The determination processing unit, For each of the multiple verification areas set above, one or more of the following image quality conditions are determined: normal, blurry, out of focus, slightly out of focus. If the number of verification areas where the image quality is determined to be blurry or out of focus is more than half or a large number, the system will prompt or have the system retake the image data. The information processing system according to feature 5.

8. The determination processing unit, For each of the multiple verification areas set above, one or more of the following image quality conditions are determined: normal, blurry, out of focus, slightly out of focus. If the number of verification areas where the image quality is determined to have slight blurring is more than half or a large number, a message will be displayed prompting the user to choose whether to store the image data as the image data to be processed or to retake the image, and the user will accept the instruction. The information processing system according to feature 5.

9. The determination processing unit, The verification area is set to one or more of the following: near the top, bottom, left, or right edges of the area detected by the area detection processing unit, or near the center. The information processing system according to feature 2.

10. The aforementioned information processing system is The aforementioned imaging processing unit includes a region detection learning processing unit that generates a learning model for detecting a predetermined region from the image data captured by the imaging processing unit, The region detection model generated by the region detection learning processing unit functions on the camera terminal. The image data captured by the aforementioned imaging processing unit is input to the region detection model to detect the predetermined region. The information processing system according to feature 1.

11. The aforementioned information processing system is The system includes a determination model generation processing unit that generates a determination model for determining whether the image data or a part thereof captured by the aforementioned imaging processing unit satisfies the image quality conditions to an extent that allows for analysis. The judgment model generated by the judgment model generation processing unit functions in the imaging terminal. The image data of the verification region is input to the judgment model to determine the image quality conditions. The information processing system according to feature 1.

12. The aforementioned information processing system is A shooting condition processing unit that determines whether the shooting conditions are met and displays a warning if it determines that the shooting conditions are not met. The information processing system according to claim 1, characterized by having the following features.

13. The aforementioned shooting condition processing unit is: The system determines whether the subject is sufficiently large to fit within the image data without extending beyond its boundaries, or whether the subject is facing the correct orientation. The information processing system according to claim 12, characterized by the features described above.

14. The aforementioned shooting condition processing unit is: Using the position of the shelf divider detected from the image data, or the length of the region detected by the region detection processing unit, and the width of the guideline in the image data, it is determined whether the subject being photographed is contained within the image data at a sufficiently large size without extending beyond it. The information processing system according to feature 13.

15. The aforementioned shooting condition processing unit is: It is determined whether the position of the gap in the display shelf detected from the image data, or the position of the region detected by the region detection processing unit, is located within a predetermined range of the image data, and, The region detection processing unit determines whether the width of the region detected is in a ratio greater than or equal to the width of the guideline. The information processing system according to feature 14.

16. The aforementioned shooting condition processing unit is: The shooting direction is determined by using a sensor provided on the shooting terminal, or by using the positional relationship of the edges of the region detected by the region detection processing unit. The information processing system according to feature 13.

17. The aforementioned shooting condition processing unit is: If it is determined that the aforementioned shooting conditions are not met, a warning will be displayed by changing the display of the area where the shooting conditions are not met. The information processing system according to claim 12, characterized by the features described above.

18. The aforementioned shooting condition processing unit is: If it is determined that the aforementioned shooting conditions are not met, the image data cannot be recorded. The information processing system according to claim 12, characterized by the features described above.

19. The aforementioned information processing system is The aforementioned determination processing unit receives image data or a portion thereof that is subject to the image quality determination process, and An identification processing unit that identifies one or more of the following from the received image data or a part thereof: The information processing system according to claim 1, characterized by having the following features.

20. A camera terminal used when taking pictures of product displays, The aforementioned shooting terminal is A shooting processing unit that photographs the sales area with the shooting device of the shooting terminal, A region detection processing unit detects a predetermined region by inputting the image data captured by the aforementioned image capture processing unit into a learning model that detects a predetermined region from the image data captured by the aforementioned image capture processing unit, A determination model for determining whether image data satisfies image quality conditions to an extent that allows for analysis is input image data of a verification region to be targeted for image quality determination processing, which is set in part or all of a predetermined region detected by the region detection processing unit, and a determination processing unit that performs image quality determination processing, A camera terminal characterized by having the following features.

21. The aforementioned shooting terminal is It has a region detection model storage unit that stores a region detection model, which is a learning model for detecting a predetermined region from the image data, The region detection processing unit, The image data captured by the aforementioned shooting processing unit is input to the region detection model to detect the predetermined region and display that region on the display device of the shooting terminal. The imaging terminal according to feature 20.

22. The aforementioned shooting terminal is The system includes a determination model storage unit that stores a determination model for determining whether the image data of the verification region satisfies predetermined image quality conditions. The determination processing unit, The image data of the verification region is input to the judgment model to determine the image quality conditions. The imaging terminal according to feature 20.

23. The aforementioned shooting terminal is A shooting condition processing unit determines whether the image data satisfies the shooting conditions, and if it determines that the shooting conditions are not satisfied, it displays a warning on the display device of the shooting terminal. The imaging terminal according to claim 20, characterized by having the following features.

24. Computers, A shooting processing unit that uses the shooting device of the shooting terminal to photograph the product sales area. A region detection processing unit that detects a predetermined region is input to a learning model that detects a predetermined region from the image data captured by the aforementioned image processing unit. A determination model for determining whether image data satisfies image quality conditions to an extent that allows for analysis is input image data of a verification region to be targeted for image quality determination processing, which is set in part or all of a predetermined region detected by the region detection processing unit, and a determination processing unit performs image quality determination processing. An information processing program characterized by functioning as such.

25. The camera terminal used when taking pictures of products in the sales area, A shooting processing unit that photographs the sales area with the shooting device of the shooting terminal, A region detection processing unit that detects a predetermined region is input to a learning model that detects a predetermined region from the image data captured by the aforementioned image processing unit. A determination model for determining whether image data satisfies image quality conditions to an extent that allows for analysis is input image data of a verification region to be targeted for image quality determination processing, which is set in part or all of a predetermined region detected by the region detection processing unit, and a determination processing unit performs image quality determination processing. A program characterized by being designed to function as such.

Images