Information processing device, information processing method, and program

Abstract

To evaluate accuracy of image recognition for different articles when different articles can be subject to image recognition in a specific environment.SOLUTION: An information processing device includes: a generation circuit that, on the basis of attribute information of first image data resulting from imaging a first article in a first environment, generates second image data that reproduces an image in which a second article differing from the first article is disposed in the first environment; an evaluation circuit that evaluates accuracy of image recognition for the second article on the basis of the second image data; and an output circuit that outputs the accuracy evaluation results.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In stores of retail stores such as supermarkets or convenience stores, a method of identifying products by image recognition (for example, a product recognition method) has been studied (for example, refer to Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in product recognition, there is room for improvement in methods for reducing the misrecognition rate of image recognition. In particular, in the recognition environment of misrecognized articles, methods for preventing misrecognition of other articles have not been conventionally studied.

[0005] Non-limiting embodiments of the present disclosure contribute to providing an information processing apparatus, an information processing method, and a program that can evaluate the accuracy of image recognition for different articles when different articles can be the target of image recognition in a specific environment.

Means for Solving the Problems

[0006] An information processing device according to one embodiment of the present disclosure comprises: a generation circuit that generates a second image data which reproduces an image in which a second item different from the first item is placed in the first environment, based on attribute information of a first image data which captures a first item in a first environment; an evaluation circuit that evaluates the accuracy of image recognition for the second item based on the second image data; and an output circuit that outputs the evaluation result of the accuracy.

[0007] These comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or recording media, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media. [Effects of the Invention]

[0008] According to one embodiment of this disclosure, when different items may be the target of image recognition in a particular environment, the accuracy of image recognition for different items can be evaluated.

[0009] Further advantages and effects of one embodiment of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects are provided by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided in order to obtain one or more identical features. [Brief explanation of the drawing]

[0010] [Figure 1] Diagram showing an example configuration of a product recognition system. [Figure 2] Flowchart showing an example of how the product recognition system works. [Figure 3] Diagram showing an example of displaying the risk of misrecognition. [Figure 4] Diagram showing an example of displaying the risk of misrecognition. [Figure 5] Diagram showing an example of displaying the risk of misrecognition. [Figure 6] Diagram showing an example of displaying the risk of misrecognition. [Figure 7]A diagram showing an example of deriving a lighting proposal. [Figure 8] Diagram showing an example of displaying the risk of misrecognition. [Figure 9] Diagram showing an example of displaying the risk of misrecognition. [Figure 10] A diagram showing an example of a computer hardware configuration. [Modes for carrying out the invention]

[0011] Each embodiment of this disclosure will be described in detail below with reference to the drawings.

[0012] For example, a system that uses image recognition to recognize products at checkout counters (or cash registers) in retail stores (for example, called a "product recognition system") is being considered. Note that "product" is just one example of "goods." For instance, the target of image recognition is not limited to products.

[0013] In image recognition, for example, lighting conditions can vary from store to store. For instance, if the lighting environment differs from the environment in which the product model is trained (referred to as the "training environment"), misrecognition of products (including failure to recognize them) may occur.

[0014] If the cause of misidentification of a product is a difference between the learning environment and the environment in which the product is recognized (for example, referred to as the "recognition environment"), then it is possible that misidentification may occur for other products in the same recognition environment. More specifically, if a shadow or similar effect on a product in the recognition environment is the cause of misidentification, then it is highly likely that similar shadows will occur for other products as well. Therefore, when misidentification occurs for a certain product, it may be desirable to evaluate (or estimate) the risk (for example, the "misidentification risk") of misidentification occurring for other products in the recognition environment in which the misidentification of that product occurred. In other words, it may be desirable to evaluate whether misidentification of other products is likely to occur in the environment in which misidentification occurred for a certain product.

[0015] Here, in the evaluation of the misrecognition risk for a product where misrecognition has occurred and another product, it may be difficult to faithfully reproduce the past recognition environment (e.g., lighting environment) where the misrecognition occurred, and it may also be difficult to evaluate the misrecognition risk of another product at the timing when the misrecognition occurred.

[0016] In one embodiment of the present disclosure, for example, a method for determining (or evaluating) the misrecognition risk for a product different from the product where misrecognition has occurred in the recognition environment where the misrecognition has occurred will be described.

[0017] (Embodiment 1) [Configuration of Product Recognition System] FIG. 1 is a diagram showing a configuration example of a product recognition system 1 according to the present embodiment.

[0018] The product recognition system 1 shown in FIG. 1 may include, for example, an image acquisition unit 11, a product recognition unit 12, a storage unit 13, a misrecognition analysis unit 14, and a result display unit 15. The misrecognition analysis unit 14 may be read as another name such as, for example, a "misrecognition determination device", an "image recognition device", an "image processing device", or an "information processing device".

[0019] The image acquisition unit 11 acquires, for example, image data (e.g., product image data) including a product at a location (e.g., a checkout counter) where image recognition is performed by the product recognition unit 12. For example, the image acquisition unit 11 may acquire product image data obtained by imaging a product existing at a location where image recognition is performed at the time of checkout with a camera (not shown). The acquired product image data is transmitted, for example, from the image acquisition unit 11 to the product recognition unit 12.

[0020] The product recognition unit 12 may, for example, perform image recognition on a product that a customer in a store checks out (e.g., purchases). The product recognition unit 12 may recognize a product corresponding to the product image data input from the image acquisition unit 11 based on a learned recognition model of the product (hereinafter referred to as a "product recognition model") stored in the storage unit 13. Information indicating the recognition result of the product may be output, for example, from the product recognition unit 12 to the result display unit 15.

[0021] Furthermore, the product recognition unit 12 may determine, for example, whether or not a misrecognition has occurred in the product recognition result. For example, the product recognition unit 12 may detect the occurrence of a misrecognition based on the user's (e.g., store clerk or customer) judgment at the time of payment. More specifically, the product recognition unit 12 may detect that a misrecognition has occurred when the user inputs that a misrecognition has occurred. The product recognition unit 12 may also detect the occurrence of a misrecognition at a time other than at the time of payment. For example, when aggregating store sales data, the product recognition unit 12 may detect that a misrecognition has occurred if there is a discrepancy between the actual inventory and the theoretical sales. In addition, if a misrecognition occurs, the product image data of the misrecognized product may be output from the product recognition unit 12 to the misrecognition analysis unit 14.

[0022] The memory unit 13 stores, for example, a product recognition model. The product recognition model may include, for example, product image data and product information such as the product name.

[0023] The misrecognition analysis unit 14 evaluates, for example, the accuracy of image recognition for products. As an example of an indicator related to accuracy, an indicator showing the possibility of misrecognition (hereinafter referred to as "misrecognition risk") may be determined (or judged or evaluated) by the misrecognition analysis unit 14.

[0024] For example, the misrecognition analysis unit 14 may generate image data including another product in the recognition environment (e.g., the first environment) based on the attribute information (e.g., also simply called "attributes") of image data of a product photographed in the recognition environment (e.g., the first environment) of the product recognition unit 12, and evaluate the accuracy of image recognition for the other product based on the generated image data.

[0025] Attributes in image data are, for example, information about the features within the image, and are information that, when changed, changes the features of the image. For example, non-exclusive examples of attributes set in product recognition system 1 include the direction of lighting, the intensity of lighting, the color temperature of the lighting, the presence or absence of blur, the orientation of the object to be recognized (e.g., a product), the intensity of reflection, and individual differences in the object to be recognized. Note that the types of attributes are not limited to these and may include, for example, other features related to product recognition.

[0026] For example, the misrecognition analysis unit 14 may acquire product image data (hereinafter also referred to as "misrecognized image data") of products that have been determined to be misrecognized by the product recognition unit 12 in image recognition. The misrecognition analysis unit 14 may also modify the attribute information of product image data of products different from the product determined to be misrecognized (for example, also referred to as "evaluation image data") based on the attribute information of the misrecognized image data. The misrecognition analysis unit 14 may then evaluate the accuracy of image recognition for the product based on the image data after the attribute information has been modified (hereinafter referred to as "pseudo-evaluation image data").

[0027] The misrecognition analysis unit 14 may, for example, output information regarding the evaluation results of the image recognition accuracy to the results display unit 15.

[0028] The result display unit 15 (or output unit) may, for example, display (or output) the recognition result of the product input from the product recognition unit 12. The result display unit 15 may also, for example, display (or output) information regarding the misrecognition risk determined by the misrecognition analysis unit 14. An example of how the misrecognition risk is displayed in the result display unit 15 will be described later.

[0029] Furthermore, at least one of the image acquisition unit 11, product recognition unit 12, storage unit 13, and result display unit 15 may or may not be included in the misrecognition analysis unit 14.

[0030] [Example configuration of the misrecognition analysis unit 14] The misrecognition analysis unit 14 may include, for example, a performance evaluation image database (DB) 141, an image encoding unit 142, an attribute manipulation unit 143, an image generation unit 144, and an evaluation unit 145. For example, the attribute manipulation unit 143 and the image generation unit 144 may correspond to a generation circuit.

[0031] The performance evaluation image database 141 stores, for example, image data of products that may be handled in a store (e.g., evaluation image data).

[0032] The image encoding unit 142 encodes, for example, misrecognized image data input from the product recognition unit 12. The image encoding unit 142 also encodes, for example, at least one evaluation image data stored in the performance evaluation image DB 141. The image encoding unit 142 may encode the misrecognized image data and the evaluation image data based, for example, on a product image encoding model. The image encoding unit 142 outputs, for example, encoded information (for example, a code relating to at least one attribute) of the misrecognized image and the evaluation image to the attribute manipulation unit 143.

[0033] Examples of product image coding models (or coding methods) include, for instance, a method that obtains a real vector of a certain dimension by convolution of images, similar to a deep learning encoder, or a method that generates an image using an image generation model from a code (e.g., a real vector) and searches for a code that is closer to the misrecognized image.

[0034] The attribute manipulation unit 143 may, for example, manipulate attribute information in the encoded information of evaluation image data input from the image encoding unit 142. The attribute manipulation unit 143 may also, for example, manipulate attributes in the encoded information of misrecognized image data input from the image encoding unit 142. The attribute manipulation unit 143 may, for example, perform attribute manipulation based on a generated image attribute manipulation model. Note that the term "manipulation" may be replaced with other terms such as "adjustment" or "modification."

[0035] Furthermore, the attribute manipulation unit 143 may, for example, apply (or reflect) attribute information related to the lighting environment (e.g., lighting conditions) (e.g., the first attribute) from the attribute information included in the encoded information of the misrecognized image data to the evaluation image data. For example, the attribute manipulation unit 143 may mix the attribute information related to the lighting environment of the misrecognized image data with the attribute information related to the lighting environment of the evaluation image data, or it may replace the attribute information related to the lighting environment of the evaluation image data with the attribute information related to the lighting environment of the misrecognized image data.

[0036] Furthermore, the attribute manipulation unit 143 may adjust, for example, at least one of the following attributes to the evaluation image data: attribute information related to the lighting environment (e.g., a first attribute) and an attribute related to a feature quantity different from the lighting environment (e.g., a second attribute related to the object to be recognized). The second attribute may be an attribute related to the surrounding environment that is different from both the lighting and the object.

[0037] The attribute manipulation unit 143 outputs, for example, the encoded information of the evaluation image after attribute manipulation to the image generation unit 144.

[0038] The image generation unit 144 generates (in other words, decodes) image data (e.g., pseudo-evaluation image data) based on the encoded information of the evaluation image input from the attribute manipulation unit 143. The image generation unit 144 may generate images based on, for example, a product image generation model. The image generation unit 144 may generate pseudo-evaluation image data using, for example, a generator in deep learning. The image generation unit 144 outputs the generated pseudo-evaluation image data to the evaluation unit 145.

[0039] The evaluation unit 145 may evaluate the accuracy of product recognition for the pseudo-evaluation image data input from the image generation unit 144 based on the product recognition model stored in the storage unit 13. For example, the evaluation unit 145 may determine the risk of misrecognition of the product being evaluated corresponding to the pseudo-evaluation image data based on the product recognition result. The evaluation unit 145 may, for example, output information regarding the evaluation result for the pseudo-evaluation image data to the result display unit 15.

[0040] [Example of operation of product recognition system 1] Next, we will describe an example of the operation of the product recognition system 1 mentioned above.

[0041] Figure 2 is a flowchart showing an example of the operation of the misrecognition analysis unit 14 of the product recognition system 1. The misrecognition analysis unit 14 may, for example, perform the processing shown in Figure 2 when the product recognition unit 12 detects that a product misrecognition has occurred.

[0042] In Figure 2, the misrecognition analysis unit 14 encodes, for example, the misrecognition image data input from the product recognition unit 12 (S101).

[0043] The misrecognition analysis unit 14 may, for example, perform the following S102 to S107 processes related to the evaluation of recognition accuracy on at least one evaluation image (in other words, at least one image of a product), and may also perform at least one attribute operation (for example, an attribute operation condition). For example, the product to be evaluated for recognition accuracy may be predetermined or selected by accepting input from the user. Also, the attribute operation conditions applied in the recognition accuracy evaluation may be predetermined or selected by accepting input from the user.

[0044] The misrecognition analysis unit 14, for example, encodes the evaluation image data (S102).

[0045] The misrecognition analysis unit 14 may, for example, perform at least one attribute operation (e.g., attribute conversion or adjustment) on the encoded information of the misrecognized image data (S103).

[0046] The misrecognition analysis unit 14 may, for example, perform at least one attribute operation on the encoded information of the evaluation image data (S104).

[0047] For example, the misrecognition analysis unit 14 may perform predetermined attribute operations on at least one of the misrecognized image and the evaluation image. Attribute operations may include, for example, operations on attributes such as the direction of lighting, the intensity of lighting, the color temperature of lighting, the intensity of reflection, the orientation of the item being evaluated (e.g., a product), or individual differences in the item.

[0048] For example, in the processing of S103, the misrecognition analysis unit 14 may adjust attributes related to the lighting environment (e.g., lighting direction, intensity, or color temperature) to the encoded information of the misrecognized image data. This adjustment makes it possible to simulate an environment that is darker or brighter than the actual lighting environment.

[0049] Furthermore, for example, in the processing of S104, the misrecognition analysis unit 14 may adjust the encoded information of the evaluation image data with respect to attributes related to the item being evaluated (for example, the orientation of the item or individual differences). This adjustment allows for the reproduction of various states of the product.

[0050] Furthermore, the misrecognition analysis unit 14 does not have to perform attribute manipulation processing in either or both of S103 and S104.

[0051] The misrecognition analysis unit 14, for example, reflects information about the misrecognized image data in the information about the evaluation image data (S105). For example, the misrecognition analysis unit 14 may reflect attribute information related to the lighting environment for image recognition (or "lighting environment information"), which is included in the encoded information of the misrecognized image data, into the encoded information of the evaluation image data (for example, lighting environment information). The term "reflect" may be replaced with other terms such as "update," "rewrite," "set," "apply," or "mix." For example, the misrecognition analysis unit 14 may perform a process such as styleMixing between the lighting environment information (or code) of the misrecognized image data and the lighting environment information (or code) of the evaluation image data.

[0052] The misrecognition analysis unit 14 generates pseudo-evaluation image data based on the encoded information of the evaluation image data, which reflects the lighting environment information of the misrecognized image data (S106). In other words, the evaluation image data is converted into pseudo-evaluation image data based on the misrecognized image data.

[0053] The misrecognition analysis unit 14 evaluates, for example, the recognition accuracy (e.g., recognition success rate or misrecognition risk) for the product corresponding to the evaluation image data based on the pseudo-evaluation image data (S107).

[0054] The misrecognition analysis unit 14 outputs the evaluation result of the recognition accuracy to the result display unit 15, for example (S108).

[0055] [Method for generating pseudo-evaluation image data] The following describes an example of a method for generating pseudo-evaluation image data.

[0056] <Generation method 1> In generation method 1, pseudo-evaluation image data may be generated by performing a predetermined image processing on the entire evaluation image data, for example. For example, the misrecognition analysis unit 14 may generate pseudo-evaluation image data by performing image processing on the evaluation image data, such as gamma correction or the addition (or removal) of Gaussian noise.

[0057] For example, the misrecognition analysis unit 14 may generate pseudo-evaluation image data that simulates the lighting environment corresponding to the misrecognized image data by adjusting the gamma correction value or Gaussian noise value in the misrecognized image data (for example, attribute information related to the brightness of the misrecognized image data).

[0058] Method 1 allows for the generation of pseudo-evaluation image data using a simple procedure, thereby reducing the processing load (or processing burden) associated with generating pseudo-evaluation image data.

[0059] <Generation method 2> In generation method 2, for example, the misrecognition analysis unit 14 may generate pseudo-evaluation image data using a 3D model. For example, the misrecognition analysis unit 14 may generate a 3D model of the item to be evaluated (e.g., a product) and generate pseudo-evaluation image data with controlled lighting environment in the 3D model based on the recognition environment corresponding to the misrecognition image data (e.g., attribute information regarding the recognition environment that was determined to be misrecognized).

[0060] For example, the misrecognition analysis unit 14 may reproduce in a 3D model the lighting environment, such as the number, type, location, or intensity of lights, and the surrounding environment, such as the location, color, and material of objects (e.g., walls or mirrors) around the location where product recognition is performed, based on the recognition environment corresponding to the misrecognized image data. In addition, the misrecognition analysis unit 14 may reproduce in a 3D model the product that is the subject of recognition accuracy evaluation, for example, based on evaluation image data.

[0061] In generation method 2, for example, to reproduce the lighting environment, a 3D model of objects surrounding the location where product recognition is performed is generated in addition to the lighting conditions. Furthermore, at least one of the light source positions and light intensity of multiple real-world lighting sources is reproduced. In addition, to reproduce individual differences in products, separate 3D models are generated for each object to be recognized, even for identical products. Therefore, if this data can be accurately collected, generation method 2 can reproduce high-quality pseudo-evaluation image data that accurately reflects the surrounding environment, light sources, and objects to be recognized.

[0062] <Generation method 3> In generation method 3, for example, the misrecognition analysis unit 14 encodes the evaluation image data and generates pseudo-evaluation image data by reflecting the attributes obtained from encoding the misrecognition image data. The misrecognition analysis unit 14 may use a Generative Adversarial Network (GAN) to transform the evaluation image data (in other words, generate pseudo-evaluation image data) when reflecting these attributes. GAN is one of the image generation technologies based on machine learning models that use neural networks such as deep learning. In this embodiment, the misrecognition analysis unit 14 uses a GAN to generate pseudo-evaluation image data, for example. In other words, in this embodiment, the GAN may be used to generate evaluation data rather than to generate training data in machine learning.

[0063] The misrecognition analysis unit 14 may generate pseudo-evaluation image data by, for example, adjusting the setting value (e.g., feature quantity) of at least one attribute of the evaluation image data based on the attributes of the misrecognized image data. For example, the misrecognition analysis unit 14 may pseudo-change the lighting environment in the evaluation image data by adjusting the attribute related to the lighting environment. Also, for example, the misrecognition analysis unit 14 may pseudo-change the orientation of an item in the evaluation image data by adjusting the attribute related to the orientation of the item. In addition, the misrecognition analysis unit 14 may not only reflect the attributes of the misrecognized image data in the evaluation image data, but may also apply arbitrary attributes such as orientation or lighting environment specified by the user to the evaluation image data.

[0064] In generation method 3, for example, multiple attributes can be changed individually, allowing for more precise adjustment of the lighting environment for evaluation image data compared to generation method 1. Furthermore, generation method 3 allows for the modification of attributes such as the shooting angle (e.g., the orientation of the product), the direction of the lighting, or individual differences in the recognized object, thus enabling the reproduction of conditions that cannot be reproduced in generation method 1.

[0065] Furthermore, in generation method 3, pseudo-evaluation image data can be generated by manipulating attributes related to the lighting environment (for example, some codes in the encoded information). Also, in generation method 3, pseudo-evaluation image data can be generated by attribute manipulation without training data. Therefore, in generation method 3, pseudo-evaluation image data can be generated without preparing a huge amount of data. Thus, compared to generation method 2, generation method 3 can easily evaluate the accuracy of image recognition for products different from those determined to be misrecognized, and the amount of processing (or processing load) for generating pseudo-evaluation image data can be reduced.

[0066] Furthermore, the machine learning model used to generate pseudo-evaluation image data in generation method 3 is not limited to GANs; other models may also be used.

[0067] [Example of displaying the risk of misrecognition] The misrecognition analysis unit 14 may output a signal to the results display unit 15 for displaying the evaluation result of the misrecognition risk (or an index related to recognition accuracy) of products corresponding to multiple evaluation image data. For example, in the misrecognition analysis unit 14, the signal for displaying the evaluation result may be output from the evaluation unit 145 to the results display unit 15, or it may be output from an output unit (not shown) to the results display unit 15.

[0068] The results display unit 15 may display the evaluation results of the product misrecognition risk (or an indicator related to recognition accuracy) based on the signal from the misrecognition analysis unit 14. The display method is not limited. As an example of a non-limiting method, a graph display or a list display may be applied.

[0069] The following is an example of how to display the risk of misrecognition.

[0070] <Example 1> Figures 3, 4, 5, and 6 show examples of the display screen for misrecognition risk in Display Example 1.

[0071] The screens shown in Figures 3, 4, 5, and 6 may include, for example, an area for selecting conditions related to recognition performance evaluation (e.g., "Recognition Performance Evaluation Condition Selection" area), an area for displaying the results of the recognition performance evaluation (e.g., "Recognition Performance Evaluation Results" area), and an area for displaying the proposed lighting environment (e.g., "Lighting Proposal Results" area).

[0072] The "Recognition Performance Evaluation Condition Selection" area may include, for example, an area to prompt the user to select a misrecognized image that reproduces lighting conditions. A signal (which may be called an operation signal) corresponding to the user's operation on the screen (for example, a selection operation such as touch or click) is input to the result display unit 15 (or the misrecognition analysis unit 14), and processing (which may include, for example, display control) corresponding to the operation signal is executed.

[0073] Furthermore, the "Recognition Performance Evaluation Condition Selection" area may include, for example, an area for selecting an evaluation image to display the results of the recognition performance evaluation. The evaluation images to display the results of the recognition performance evaluation may include, for example, an evaluation image stored in the performance evaluation image DB141 (for example, referred to as the "original performance evaluation image"), and an evaluation image to which the lighting environment conditions in which misrecognition occurred (for example, referred to as the "misrecognition conditions") have been applied to the evaluation image stored in the performance evaluation image DB141 (for example, referred to as the "misrecognition condition applied image").

[0074] Furthermore, the images to which the misrecognition conditions are applied may include, for example, evaluation image data to which attribute operations other than the misrecognition conditions are applied. In the examples shown in Figures 3 to 6, the attributes that can be selected as attribute operations include the orientation of the target object, individual differences of the target object, the direction of the lighting, the intensity of the lighting, the color temperature of the lighting, and the intensity of the reflection. Note that the selectable attributes are not limited to these, and other attributes may also be used. For example, if a certain attribute operation is selected, the misrecognition analysis unit 14 may apply a predetermined set value (also called the attribute operation intensity) to that attribute operation, or it may determine the set value by accepting input from the user (for example, input from another screen).

[0075] Furthermore, the "Recognition Performance Evaluation Condition Selection" area may include, for example, an area to prompt the user to choose (or decide) whether or not to implement the lighting environment suggestion.

[0076] For example, the user may select misrecognized image data that reproduces the misrecognition conditions (file name: false_images in Figures 3 to 6) and the evaluation image to be evaluated for recognition performance on the screen, and then press the "Execute" button. Alternatively, for example, the user may select to execute the lighting environment proposal on the screen and then press the "Execute" button. When the misrecognition analysis unit 14 detects that the user has pressed the execute button, it may start processing related to recognition performance evaluation (for example, generating pseudo-evaluation image data and performing recognition performance evaluation).

[0077] As shown in Figures 3 to 6, the evaluation results of recognition performance for evaluation images can be, for example, information indicating the probability of successfully recognizing the corresponding product in the evaluation image data in product recognition (e.g., "recognition rate" or recognition accuracy rate). For example, the lower the recognition rate, the higher the possibility of misrecognition of the product in question (in other words, the risk of misrecognition or recognition failure rate).

[0078] The recognition rate may be calculated, for example, by determining whether the result obtained when the pseudo-evaluation image data is recognized by the product recognition model matches the correct product. For example, the misrecognition analysis unit 14 may set a high recognition rate if the product evaluated by the product recognition model as having the highest similarity to the pseudo-evaluation image data matches the correct product. Conversely, if the product evaluated by the product recognition model as having the highest similarity to the pseudo-evaluation image data does not match the correct product, the misrecognition analysis unit 14 may set a low recognition rate because there is a risk of misrecognizing it as an incorrect product.

[0079] Furthermore, even if the product evaluated as having the highest similarity matches the correct product, if the recognition scores for other products are also high, there is a risk that it may be misrecognized as another product with only slight changes in the environment. For this reason, the misrecognition analysis unit 14 may set a higher misrecognition rate when the similarity is high for multiple products than when the similarity is high only for the correct product. Also, if the similarity is low for all products (for example, below a predetermined threshold), it is difficult for the product recognition model to recognize the product itself, so the misrecognition analysis unit 14 may set a lower recognition rate.

[0080] Furthermore, the evaluation results for recognition performance are not limited to the recognition rate; other parameters may also be used. Examples of other parameters include information indicating the probability of failing to recognize a product (e.g., "misrecognition rate") or the similarity score mentioned above.

[0081] The misrecognition analysis unit 14 may, for example, determine whether the recognition rate of each product (also called a class) is higher than a predetermined threshold (recognition rate: 95% in Figures 3 to 6). For example, information regarding products with a recognition rate lower than the threshold (e.g., the number of failing classes) may be displayed on the screen. In the examples of Figures 3 to 6, the misrecognition analysis unit 14 may output a signal to the results display unit 15 for displaying the evaluation results (e.g., recognition rate) for each product as a graph. The graph display allows the user to, for example, cross-sectionally check the recognition performance of multiple products to which misrecognition conditions have been applied.

[0082] Furthermore, as shown in Figures 4 and 5, for example, the misrecognition analysis unit 14 may output a signal to the results display unit 15 for highlighting (for example, by coloring or patterning) the graph corresponding to products whose recognition rate is below a threshold. This highlighting makes it easier for the user to identify products that are likely to be misrecognized.

[0083] Furthermore, as shown in Figures 3 to 6, when a graph of a particular product (for example, the graph for product A in Figures 3 to 6) is selected by an operation such as mouseover or click, information about the product, such as the product name and recognition rate, may be displayed.

[0084] Furthermore, the recognition rate for each product may be displayed not only as a graph but also as a list or other format. Also, the threshold is not limited to 95% and may be set to other values. For example, the threshold may be determined based on the recognition accuracy required for the operation of product recognition system 1. Additionally, the threshold may be set to be variable.

[0085] Furthermore, the misrecognition analysis unit 14 is not limited to highlighting, but may also output a signal to the result display unit 15 for distinguishing and displaying information about products whose recognition rate exceeds a predetermined threshold from information about products whose recognition rate does not exceed a predetermined threshold.

[0086] The following examples of display screens are explained using Figures 3 to 6. In these examples, the user can input information for "Selection of misrecognized image to reproduce lighting conditions," "Selection of recognition performance evaluation image," and "Selection of lighting suggestion." Here, "Selection of misrecognized image to reproduce lighting conditions" is an input section that accepts the selection of an image to extract the lighting conditions that should be reflected in the performance evaluation image for each product. "Selection of recognition performance evaluation image" is an input section that accepts the selection of whether to use the original image (original performance evaluation image) or an image to which the misrecognition conditions have been applied (misrecognition condition applied image) when evaluating recognition performance. If "misrecognition condition applied image" is selected, the user can further select each attribute to apply or modify. "Selection of lighting suggestion" is an input section that accepts the selection of whether or not to request automatic lighting suggestions that reduce the likelihood of misrecognition.

[0087] The example display screen shown in Figure 3 is an example where the user selects the original image for performance evaluation, does not select the image with misrecognition conditions applied, and does not select the lighting suggestion. Note that since Figure 3 is an example where the attributes of the misrecognition image are not applied to the performance evaluation image, in Figure 3, for example, the misrecognition image that reproduces the lighting conditions does not need to be selected.

[0088] In the case of Figure 3, the misrecognition analysis unit 14 may, for example, evaluate the recognition rate of the original performance evaluation image without applying the misrecognition conditions based on the misrecognized image to the original performance evaluation image.

[0089] The product recognition model uses a model that performed well in evaluations using, for example, the original image used for performance evaluation. Therefore, the recognition rate for the original image tends to be higher than the specified threshold (e.g., 95%). Consequently, as shown in Figure 3, for example, the evaluation result of the recognition performance for the original image may show that the recognition rate of all products is higher than the threshold (95%), and there are zero products in the failing class.

[0090] The example display screen shown in Figure 4 illustrates a scenario where the user does not select the original image for performance evaluation, instead selecting the image with misrecognition conditions applied, and not selecting the lighting suggestion. In Figure 4, for example, a misrecognition image (filename: false_images) that reproduces the lighting conditions is selected. Note that in Figure 4, it is assumed that the attributes of the misrecognition image are applied as is, so no individual checks are performed on "attribute manipulation".

[0091] In the case of Figure 4, the misrecognition analysis unit 14 may, for example, apply misrecognition conditions (e.g., lighting environment information) based on the selected misrecognition image to the original image for performance evaluation to generate pseudo-evaluation image data, and then evaluate the product recognition performance (e.g., recognition rate) of the pseudo-evaluation image data.

[0092] In the example shown in Figure 4, the evaluation results for recognition performance on simulated evaluation image data show that the recognition rate of three products, including product A, is below the threshold (95%), and that there are three products in the failing class. The display screen shown in Figure 4 allows the user to cross-reference the recognition performance (e.g., misrecognition risk) of multiple products under the same lighting environment as the product corresponding to the misrecognized image. In the example shown in Figure 4, the user can see that the recognition rate of three products, including product A, is below the threshold, and that the misrecognition risk has increased.

[0093] The example display screen shown in Figure 5 is an example where the user does not select the original image for performance evaluation, but instead selects the image with misrecognition conditions applied and does not select the lighting suggestion. In Figure 5, as an example, the attribute operation "lighting intensity" is selected. Also in Figure 5, for example, a misrecognition image that reproduces the lighting conditions (file name: false_images) is selected. As a result, Figure 5 displays the misrecognition risk when the selected attribute (lighting intensity in Figure 5) of the misrecognition image is reproduced.

[0094] In the case of Figure 5, the misrecognition analysis unit 14 may, for example, apply attribute manipulation of illumination intensity and misrecognition conditions (e.g., illumination environment information) based on the selected misrecognition image to the original image for performance evaluation to generate pseudo-evaluation image data, and then evaluate the product recognition performance (e.g., recognition rate) of the pseudo-evaluation image data.

[0095] In the example shown in Figure 5, the evaluation results for recognition performance on simulated evaluation image data show that the recognition rate of four products, including product A, is below the threshold (95%), and that there are four products in the failing class. The display screen shown in Figure 5 allows the user to cross-reference the recognition performance (e.g., misrecognition risk) of multiple products under conditions different from the lighting environment (misrecognition conditions) for the product corresponding to the misrecognition image, as well as under different conditions (changes in lighting intensity in Figure 5). In the example shown in Figure 5, the user can check the misrecognition risk (e.g., potential misrecognition risk) when the lighting intensity is further changed under the misrecognition conditions.

[0096] Note that while Figure 5 shows an example where one attribute operation is selected, multiple attribute operations may also be selected.

[0097] In Figures 3, 4, and 5, the selection of recognition performance evaluation images is explained in the case where either the original performance evaluation image or the image with misrecognition conditions applied is selected. However, the system is not limited to this, and both the original performance evaluation image and the image with misrecognition conditions applied may be selected. In this case, for example, the graph display when the original performance evaluation image shown in Figure 3 is selected and the graph display when the image with misrecognition conditions applied is selected, as shown in Figure 4 or Figure 5, may be displayed. For example, the recognition performance evaluation results shown in Figure 3 and the recognition performance evaluation results in Figure 4 or Figure 5 may be displayed side by side on the screen, displayed individually by switching screens, or overlaid in a distinguishable manner.

[0098] This allows users to recognize the differences in the risk of misrecognition depending on whether or not misrecognition conditions are applied, and whether or not additional conditions are applied through attribute manipulation.

[0099] The screen shown in Figure 6 is an example where the user does not select the original image for performance evaluation, but instead selects the image with misrecognition conditions applied and then selects the lighting suggestion. In Figure 6, for example, a misrecognition image (filename: false_images) that reproduces the lighting conditions is selected. In Figure 6, the risk of misrecognition is displayed when the lighting conditions based on the misrecognition image are reproduced and then the lighting corresponding to the lighting suggestion is also applied.

[0100] In the case of Figure 6, the misrecognition analysis unit 14 may, for example, apply misrecognition conditions (e.g., lighting environment information) based on the selected misrecognition image to the original image for performance evaluation to generate pseudo-evaluation image data, and then evaluate the product recognition performance (e.g., recognition rate) of the pseudo-evaluation image data.

[0101] For example, the evaluation results may show that some products have a recognition rate below a threshold. The misrecognition analysis unit 14 may, for example, manipulate an attribute related to lighting (for example, at least one of the lighting direction, intensity, and color temperature) to determine (for example, search for) candidate setting values ​​for an attribute that will make the recognition rate of multiple products (for example, a specified number of products, or all products) higher than the threshold, and output a signal to the result display unit 15 for displaying the lighting proposal result including the determined candidate setting values ​​(for example, the proposed values). In the example shown in Figure 6, the evaluation result of the recognition performance for the evaluation image in the proposed lighting environment shows that all recognition rates are higher than the threshold (95%) and there are 0 failing classes. Also in the example shown in Figure 6, the results related to the proposed lighting environment (for example, attribute setting values) are displayed for each of the lighting direction, intensity, and color temperature.

[0102] As shown in Figure 6, the display screen allows users to check lighting adjustment methods (for example, adding lighting, or adjusting the direction, intensity, or color temperature of the lighting) that reduce the risk of misrecognition, for lighting environments similar to the lighting environment (misrecognition conditions) for the product corresponding to the misrecognized image.

[0103] Furthermore, the conditions for lighting suggestions are not limited to conditions where the recognition rate of multiple (e.g., all) products exceeds a threshold. For example, the conditions may also include conditions where the number of products whose recognition rate due to attribute manipulation related to lighting is below the threshold is smaller (e.g., the minimum or below the threshold). Also, for example, if there are multiple attribute operations (e.g., combinations of attribute operations) that result in a recognition rate of above the threshold, the misrecognition analysis unit 14 may apply the attribute operation with the highest average recognition rate among the multiple products (e.g., the highest) to the lighting suggestion result.

[0104] Furthermore, the misrecognition analysis unit 14 may, for example, make lighting suggestions that increase the recognition rate of a particular product based on user specifications, product price, sales figures, or advertising content. For example, high-priced products have a greater impact when misrecognition occurs compared to other products. Also, popular products or products advertised are targeted for product recognition more frequently, so a low recognition rate will result in a large number of misrecognitions. Therefore, even if the recognition rate of other products decreases slightly, making lighting suggestions that improve the recognition rate of these products can reduce the negative impact on overall store sales and operations.

[0105] Furthermore, in Figure 6, the parameters that determine the attribute operations of the lighting suggestions by the misrecognition analysis unit 14 (for example, candidate attribute setting values) may be selected by the user (in other words, a manual method), or they may be predetermined by the misrecognition analysis unit 14 (in other words, an automatic method).

[0106] Figure 7(a) shows an example of manual lighting environment exploration, and Figure 7(b) shows an example of automatic lighting environment exploration.

[0107] Figure 7(a) is an example of a display screen where the user can select candidates for attributes related to the lighting environment (e.g., lighting direction, lighting intensity, and lighting color temperature). For example, if the user selects a lighting suggestion in Figure 6, the display screen shown in Figure 7(a) may be accessed.

[0108] In Figure 7(a), the user may specify parameters such as the direction of lighting for the product to be evaluated (e.g., any of the 360° directions), lighting intensity (e.g., any of 0 to 100; 0 indicates off lighting, 100 indicates maximum lighting intensity), and lighting color temperature (e.g., any of the minimum to maximum values), and then press the "Execute" button. When the misrecognition analysis unit 14 detects that the "Execute" button has been pressed, it may generate pseudo-evaluation image data based on the lighting conditions selected by accepting user input, and output a signal to the results display unit 15 to display the evaluation result of the recognition performance of the pseudo-evaluation image data (e.g., the number of products with a recognition rate below a threshold (95%)). This allows the user to specify more appropriate lighting conditions (e.g., conditions with a higher recognition rate) based on the display of the recognition rate corresponding to the specified lighting conditions.

[0109] For example, the recognition rates of multiple products based on the lighting conditions determined in Figure 7(a) may be displayed graphically as shown in Figure 6.

[0110] Figure 7(b) shows examples of combinations of candidate attributes related to the lighting environment (e.g., lighting direction, lighting intensity, and lighting color temperature). In the example shown in Figure 7(b), there are 8 candidate lighting directions, 5 candidate lighting intensity, and 5 candidate lighting color temperature.

[0111] The misrecognition analysis unit 14 may, for example, generate pseudo-evaluation image data for some or all of the combinations of attributes related to the lighting environment (200 (=8 × 5 × 5) combinations in Figure 7(b)), and evaluate the recognition performance (e.g., recognition rate) of the pseudo-evaluation image data. The misrecognition analysis unit 14 may, for example, determine as the lighting suggestion result the combination in which the number of products with a recognition rate below a threshold is smaller (e.g., the minimum or below a specified value) among multiple combinations of attributes related to the lighting environment.

[0112] In this way, by displaying the lighting suggestion results, the misrecognition analysis unit 14 can prompt the user to adjust the lighting environment to reduce the risk of misrecognition. Therefore, for example, the risk of misrecognition can be reduced without performing additional learning (or retraining), which is more costly than product recognition evaluation.

[0113] Furthermore, the number of lighting suggestions determined by the misrecognition analysis unit 14 is not limited to one, but may be multiple. Also, the number of combinations (or candidates) of attributes related to the lighting environment is not limited to 200, but may be any other number. In addition, for example, at least one value among the attributes shown in Figure 7(b) (e.g., lighting direction, intensity, and color temperature) may be fixed.

[0114] Furthermore, while Figure 6 illustrates an example where no attribute operations are selected (similar to Figure 4), one or more attribute operations may be selected, as in Figure 5.

[0115] Furthermore, while Figure 6 illustrates an example of a graph displaying the recognition rate of each product based on the lighting suggestion results, the system is not limited to this. For example, the recognition rate of each product based on the lighting suggestion results, and the misrecognition rate of each product when misrecognition conditions are applied (for example, similar to Figure 4 or Figure 5) may also be displayed. This allows users to visually confirm the reduction in the risk of misrecognition through lighting suggestions.

[0116] <Example 2> In Display Example 2, for example, the misrecognition analysis unit 14 may output a signal to the result display unit 15 for displaying evaluation results (e.g., recognition rate, misrecognition risk, or recognition score described later) for new pseudo-evaluation image data generated by further adjusting at least one of the attribute information related to a feature quantity different from the said lighting environment, to pseudo-evaluation image data to which attribute information related to the lighting environment for the product corresponding to the misrecognized image has been applied.

[0117] As an example, the misrecognition analysis unit 14 may output a signal to the results display unit 15 for displaying a list of evaluation results (for example, recognition rate, misrecognition risk, or recognition score described later) corresponding to each of the multiple new pseudo-evaluation image data generated by making different changes (in other words, adjustments or attribute manipulations) to the attribute information of the evaluation image data.

[0118] Figure 8 shows an example of the display screen for misrecognition risk in Display Example 2.

[0119] The display screen shown in Figure 8 includes areas such as "Misrecognized Image," "Evaluation Image" (e.g., Registered Evaluation Image), "Attribute Manipulation," "Attribute Manipulation Strength," "Pseudo-Evaluation Image," "Recognition Score," "Recognition Result," and "Add to Training."

[0120] The "misrecognized image" area may display, for example, image data of a product that has been misrecognized by the product recognition unit 12.

[0121] The "Registered Evaluation Images" area may display, for example, an image to be evaluated from among the evaluation images registered in the performance evaluation image DB 141. For example, the products (evaluation image data) to be registered evaluation images may be determined by selection by at least one of the misrecognition analysis unit 14 and the user. For example, in Display Example 1, for products with a recognition rate below a threshold (e.g., 95%), a list of recognition scores obtained by different attribute operations may be displayed, as in Display Example 2. Alternatively, a display may be shown prompting the user to select a product to be displayed on the screen in Display Example 2 from among multiple products whose recognition rates are displayed in Display Example 1.

[0122] The "Attribute Manipulation" area may display, for example, the presence or absence of an attribute to be manipulated for the evaluation image, and the type of attribute. Figure 8 shows an example where one attribute is displayed for each evaluation image, but multiple attributes may be displayed.

[0123] The "Attribute Manipulation Strength" area may display, for example, the strength of the attribute manipulation displayed in the "Attribute Manipulation" area. The attribute manipulation strength may represent, for example, a value (level) indicating one of several candidates for the attribute manipulation strength, or it may represent the actual value set for each attribute manipulation (for example, lighting direction [degrees], lighting intensity [lx], color temperature [K]).

[0124] The attribute to be operated on, and the intensity of the attribute operation, may be selected by accepting input from the user, or they may be predetermined by the misrecognition analysis unit 14.

[0125] The "Simulated Evaluation Image" area may display simulated evaluation images to which misrecognition conditions (e.g., conditions related to the lighting environment) and attribute operations displayed in "Attribute Operation" and "Attribute Operation Intensity" have been applied to the evaluation image. For example, in Figure 8, simulated evaluation image data may be displayed to which lighting conditions corresponding to the misrecognized image of a PET bottle beverage and attribute operations corresponding to "Attribute Operation" and "Attribute Operation Intensity" have been applied. For example, in Figure 8, if the attribute operation for the evaluation image data of cup noodles is "None," simulated evaluation image data of cup noodles to which the lighting conditions corresponding to the misrecognized image of a PET bottle beverage have been applied may be displayed. Also, for example, in Figure 8, if the attribute operation for the evaluation image data of cup noodles is "Orientation Change," simulated evaluation image data of cup noodles to which the lighting conditions corresponding to the misrecognized image of a PET bottle beverage have been applied and whose orientation has been changed may be displayed. Similarly, for example, in Figure 8, if the attribute operation for the evaluation image data of cup noodles is "Reflection," simulated evaluation image data of cup noodles to which the lighting conditions corresponding to the misrecognized image of a PET bottle beverage have been applied and whose light reflection intensity has been changed may be displayed.

[0126] The "Recognition Score" area may display, for example, the evaluation result for a simulated evaluation image. As an example of an evaluation result, the recognition score, which is an index indicating the likelihood of recognizing the estimated correct class (correct product) for the simulated evaluation image, may be displayed. The misrecognition analysis unit 14 may, for example, determine a recognition score indicating the likelihood of each of multiple (for example, all) classes for the product recognition result of the simulated evaluation image. The recognition score may be, for example, a value in which the sum of the recognition scores of multiple classes equals 1 (in other words, a value normalized to the range of 0 to 1). For example, the closer the recognition score of a class is to 1, the more likely the recognition result of that class is. In other words, the closer the recognition score is to 0, the higher the risk of misrecognition.

[0127] For example, in Figure 8, the misrecognition analysis unit 14 determines the recognition scores for multiple classes (for example, multiple classes including instant noodles, confectionery, general merchandise, or beverages) in the product recognition result for the evaluation image data of instant noodles, and the recognition score for the correct class, instant noodles, among the recognition scores of the multiple classes may be displayed on the screen.

[0128] The "Recognition Result" area may display either "Incorrect Recognition" or "Correct Recognition" as the evaluation result by the misrecognition analysis unit 14. For example, the misrecognition analysis unit 14 may determine "Correct Recognition" when the recognition score of the correct class is the highest among the recognition scores of multiple classes, and determine "Incorrect Recognition" when the recognition score of a class other than the correct class is the highest.

[0129] The "Add to Learning" area may include, for example, a display prompting the user to input information to decide whether or not to perform retraining (or additional learning) using multiple pseudo-evaluation image data (e.g., adding to training data) in the list view. For example, the user may select a checkbox displayed in "Add to Learning" to decide whether to add the corresponding pseudo-evaluation image data to the training data. Alternatively, the misrecognition analysis unit 14 may add pseudo-evaluation image data whose recognition result is a misrecognition to the training data (e.g., by selecting the "Add to Learning" checkbox). Alternatively, the misrecognition analysis unit 14 may add pseudo-evaluation image data whose recognition score is below a threshold to the training data (e.g., by selecting the "Add to Learning" checkbox). By adding pseudo-evaluation image data whose recognition result is a misrecognition or whose recognition score is below a threshold to the training data as images with a high probability of misrecognition, misrecognition can be determined more accurately.

[0130] Furthermore, the user's effort may be reduced by automatically selecting the "Add to training" checkbox for pseudo-evaluation image data that meets predetermined conditions, or by providing a button that accepts instructions to automatically select the checkbox. Alternatively, the selection of training data may be automated by automatically adding pseudo-evaluation image data to the training data without requiring user confirmation. In this case, the display of the checkbox itself may be abridged. Criteria for automatic selection may include, for example, that the recognition result is a misrecognition or that the recognition score is below a threshold.

[0131] For example, if the misrecognition analysis unit 14 detects the pressing of an "Output" button (not shown), it may add the pseudo-evaluation image data selected in "Add to Learning" to the learning data.

[0132] Here, pseudo-evaluation image data with lower recognition scores for the correct class are expected to be added to the training data because the retraining effect on improving the recognition rate is greater. For example, if the "recognition score" value is below a threshold, the misrecognition analysis unit 14 may output information to the results display unit 15 to highlight the recognition score area, as shown in Figure 9. Note that the area to be highlighted is not limited to the recognition score, but may also include other areas of the relevant pseudo-evaluation image data. By highlighting, the misrecognition analysis unit 14 can prompt the user to add the relevant pseudo-evaluation image data to the training data.

[0133] Furthermore, pseudo-evaluation image data that has been judged as "correctly recognized" can be added to the training data as ground truth data. Generally, the higher the recognition rate, the harder it is to improve, so if the threshold for judging as "correctly recognized" is sufficiently high, the effect will be minimal, but even in this way, an improvement in accuracy can be expected.

[0134] As shown in Example 2, for recognition targets that may be misrecognized, candidate images to add to the training data are displayed. This makes it easier for users to add images with a higher risk of misrecognition (e.g., images with lower recognition scores) to the training data, thereby improving product recognition accuracy and reducing the risk of misrecognition through additional training.

[0135] Furthermore, for example, pseudo-evaluation image data with a high risk of misrecognition is more likely to be added to the training data. Therefore, pseudo-evaluation image data with a low risk of misrecognition is less likely to be added to the training data. This can suppress the increase in product image data added to the training process, which has a higher processing cost compared to product recognition evaluation, and improve the efficiency of retraining.

[0136] Note that the display content shown in Figures 8 and 9 is just an example and is not limited to these. For example, some of the information shown in Figures 8 and 9, such as the misrecognized image, registered evaluation image, attribute manipulation, attribute manipulation intensity, pseudo-evaluation image, recognition score, recognition result, and additional information for learning, may be displayed, or other information may be displayed in addition to some of the information.

[0137] The above explains examples of how to display the risk of misrecognition.

[0138] In this way, the misrecognition analysis unit 14, for example, based on the attribute information of the misrecognized image data of the product taken in the environment in which the misrecognition of the product occurred, generates a pseudo-image data of another product in the environment in which the misrecognition occurred (for example, the lighting environment) (in other words, it reproduces an image with a different product placed there), evaluates the accuracy of image recognition for the other product (for example, the risk of misrecognition), and outputs a signal to display (visualize) the evaluation result of the image recognition accuracy.

[0139] As a result, the misrecognition analysis unit 14 can determine (or evaluate) the risk of misrecognition for products different from the product that was misrecognized in an environment where misrecognition of a product occurred. Therefore, the misrecognition analysis unit 14 can evaluate the accuracy of image recognition for different items when different items may be the target of image recognition in a particular environment.

[0140] For example, in addition to products that were misrecognized in lighting environments different from the learning environment, users can also check the risk of misrecognition in lighting environments where misrecognition occurred for other products, and take measures such as additional learning and adjustment of the lighting environment according to the risk of misrecognition. Therefore, according to this embodiment, it is possible to prevent misrecognition of other products in the recognition environment of a misrecognized product, and to reduce the misrecognition rate of image recognition.

[0141] Furthermore, the location where product recognition is performed in the product recognition system 1 is not limited to indoors (e.g., inside a store), but may also be outdoors. For example, the misrecognition analysis unit 14 may transform the evaluation image data based on attributes related to the lighting environment (e.g., misrecognition conditions) due to at least one of the lighting environment and sunlight.

[0142] Furthermore, while Figures 3 to 6 illustrate the case of displaying the misrecognition risk (e.g., recognition rate) in a graph, the display method of the misrecognition risk is not limited to a graph; other display methods (e.g., list display) are also acceptable.

[0143] Furthermore, while Figures 3 to 6 illustrate an example where the "Recognition Performance Evaluation Condition Selection," "Recognition Performance Evaluation Results," and "Lighting Proposal Results" areas are displayed on a single screen, the system is not limited to this, and at least one of the "Recognition Performance Evaluation Condition Selection," "Recognition Performance Evaluation Results," and "Lighting Proposal Results" areas may be displayed on separate screens.

[0144] Furthermore, for example, the components included in the product recognition system 1 may be located at the location where product recognition is performed (e.g., a store). Alternatively, for example, among the components included in the product recognition system 1, the image acquisition unit 11 and the result display unit 15 may be located at the location where product recognition is performed (e.g., a store), while the other components may be located in a location physically separate from the image acquisition unit 11 and the result display unit 15 (e.g., at least one server). For example, at least one of the processes for evaluating the misrecognition risk may be performed on the server.

[0145] Furthermore, in the embodiments described above, products displayed in a store were explained as an example of an item subject to evaluation of misrecognition risk (or image recognition target), but the items subject to evaluation of misrecognition risk are not limited to products. In other words, the disclosures of this embodiment can be applied to facilities other than stores or items other than products, as long as the misrecognition risk of an item different from the item misrecognized in image recognition is evaluated under the misrecognition conditions.

[0146] Furthermore, pseudo-evaluation image data may be generated using general image processing techniques different from the generation methods 1 to 3 described in the embodiments above. Specifically, examples include transformation of product shape or orientation by geometric transformation, and transformation of saturation or color tone. In these cases, the distribution of shape or color, etc., can be extracted from the image data determined to be misrecognized and used as attributes.

[0147] Furthermore, while the above-described embodiment explained the case where attributes related to the lighting environment of image data determined to be misrecognized are applied to evaluation image data, the method is not limited to image data determined to be misrecognized. For example, attributes related to the lighting environment of image data that is not determined to be misrecognized (e.g., image data determined to be correctly recognized) may also be applied to evaluation image data. This makes it possible to reproduce image data of multiple items in various recognition environments through image processing (e.g., attribute manipulation) without taking photographs in the actual environment, thus easily evaluating the accuracy of image recognition for multiple items.

[0148] Furthermore, the notation "...part" in the above-described embodiment may be replaced with other notations such as "...circuitry," "...assembly," "...device," "...unit," or "...module."

[0149] As described above, embodiments relating to this disclosure have been described in detail with reference to the drawings, but the functions of the product management system 1 described above can be realized by a computer program.

[0150] Figure 10 shows the hardware configuration of a computer that implements the functions of each device through programming. This computer 1100 includes an input device 1101 such as a keyboard, mouse, or touchpad; an output device 1102 such as a display or speaker; a CPU (Central Processing Unit) 1103; a GPU (Graphics Processing Unit) 1104; a ROM (Read Only Memory) 1105; a RAM (Random Access Memory) 1106; a storage device 1107 such as a hard disk drive or SSD (Solid State Drive); a reader 1108 that reads information from a recording medium such as a DVD-ROM (Digital Versatile Disk Read Only Memory) or USB (Universal Serial Bus) memory; and a transceiver 1109 that communicates via a network. Each part is connected by a bus 1110.

[0151] The reading device 1108 then reads the program for realizing the functions of each of the above devices from the recording medium on which the program is stored and stores it in the storage device 1107. Alternatively, the transmitting / receiving device 1109 communicates with a server device connected to the network and stores the program for realizing the functions of each of the above devices downloaded from the server device in the storage device 1107.

[0152] Then, the CPU 1103 copies the program stored in the memory device 1107 to the RAM 1106, and sequentially reads and executes the instructions contained in that program from the RAM 1106, thereby realizing the functions of each of the above devices.

[0153] This disclosure can be implemented using software, hardware, or software integrated with hardware.

[0154] Each functional block used in the description of the above embodiments may be implemented partially or entirely as an integrated circuit (LSI), and each process described in the above embodiments may be controlled partially or entirely by a single LSI or a combination of LSIs. An LSI may consist of individual chips, or it may consist of a single chip that includes some or all of the functional blocks. An LSI may have data inputs and outputs. Depending on the degree of integration, LSIs may be referred to as ICs, system LSIs, super LSIs, or ultra LSIs.

[0155] The method of integration is not limited to LSIs; it may also be implemented using dedicated circuits, general-purpose processors, or dedicated processors. Furthermore, FPGAs (Field Programmable Gate Arrays) that can be programmed after LSI manufacturing, or reconfigurable processors that allow for the reconfiguration of the connections and settings of circuit cells within the LSI, may also be used. This disclosure may be implemented as digital or analog processing.

[0156] Furthermore, if advancements in semiconductor technology or related technologies lead to the emergence of integrated circuit technologies that replace LSIs, then naturally, these technologies can be used to integrate functional blocks. The application of biotechnology, for example, is a possible possibility.

[0157] This disclosure is applicable to all types of devices, systems, and equipment having communication capabilities (collectively referred to as communication equipment). Communication equipment may include a radio transceiver and a processing / control circuit. A radio transceiver may include a receiver and a transmitter, or both as functions. A radio transceiver (transmitter, receiver) may include an RF (Radio Frequency) module and one or more antennas. The RF module may include an amplifier, an RF modulator / demodulator, or similar. Non-exclusive examples of communication devices include telephones (mobile phones, smartphones, etc.), tablets, personal computers (PCs) (laptops, desktops, notebooks, etc.), cameras (digital still / video cameras, etc.), digital players (digital audio / video players, etc.), wearable devices (wearable cameras, smartwatches, tracking devices, etc.), game consoles, digital book readers, telehealth / telemedicine devices, vehicles or mobile transport with communication capabilities (cars, airplanes, ships, etc.), and combinations of the above-mentioned devices.

[0158] Communication devices are not limited to portable or movable devices, but also include all kinds of non-portable or fixed devices, devices, and systems, such as smart home devices (appliances, lighting equipment, smart meters or measuring instruments, control panels, etc.), vending machines, and any other "things" that may exist on an IoT (Internet of Things) network.

[0159] Furthermore, in recent years, Cyber-Physical Systems (CPS), a new concept in IoT (Internet of Things) technology that creates new added value through information linkage between the physical and cyber spaces, has been attracting attention. This CPS concept can also be adopted in the above-described embodiment.

[0160] In other words, as a basic configuration of CPS, for example, edge servers located in physical space and cloud servers located in cyberspace are connected via a network, and processing can be distributed and performed by the processors installed on both servers. Here, it is preferable that each processing data generated on the edge server or cloud server is generated on a standardized platform, and by using such a standardized platform, it is possible to improve efficiency when building systems that include various diverse groups of sensors and IoT application software.

[0161] In the above embodiment, for example, the edge server may be located in the store and perform product recognition processing and product misrecognition risk assessment processing. The cloud server may, for example, use data received from the edge server via the network to perform model training. Alternatively, for example, the edge server may be located in the store and perform product recognition processing, and the cloud server may use data received from the edge server via the network to perform product misrecognition risk assessment processing.

[0162] Communication includes data communication via cellular systems, wireless LAN systems, and communication satellite systems, as well as data communication using combinations of these.

[0163] Furthermore, the communication device also includes devices such as controllers and sensors that are connected to or linked to a communication device that performs the communication functions described in this disclosure. For example, this includes controllers and sensors that generate control signals and data signals used by the communication device that performs the communication functions of the communication device.

[0164] Furthermore, communication equipment includes infrastructure facilities such as base stations, access points, and any other devices, devices, and systems that communicate with or control the aforementioned non-limited types of equipment.

[0165] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to such examples. It is clear to those skilled in the art that various modifications or alterations can be conceived within the scope of the claims, and these will naturally also fall within the technical scope of this disclosure. Furthermore, the components in the above embodiments may be combined in any way without departing from the spirit of the disclosure.

[0166] The specific examples of this disclosure have been described in detail above, but these are merely illustrative and do not limit the scope of the claims. The technologies described in the claims include various modifications and changes to the specific examples described above.

[0167] An information processing device according to one embodiment of the present disclosure comprises: a generation circuit that generates a second image data which reproduces an image in which a second item different from the first item is placed in the first environment, based on attribute information of a first image data which captures a first item in a first environment; an evaluation circuit that evaluates the accuracy of image recognition for the second item based on the second image data; and an output circuit that outputs the evaluation result of the accuracy.

[0168] In one embodiment of the present disclosure, the evaluation result includes information indicating the probability of successfully recognizing the second item in the second image data in the image recognition, and the output circuit outputs a signal for displaying the information indicating the probability of successfully recognizing the second item in the second image data.

[0169] In one embodiment of the present disclosure, the output circuit outputs a signal for distinguishing and displaying information of a second item in the second image data where the probability of successfully recognizing the second item exceeds a predetermined threshold, and information of a second item where the probability does not exceed the predetermined threshold.

[0170] In one embodiment of the present disclosure, the generation circuit generates the second image data by changing the attribute information of the second image data of the second article based on the attribute information of the first image data.

[0171] In one embodiment of the present disclosure, the generation circuit generates the second image data by applying the first attribute information relating to the lighting environment for image recognition from the attribute information of the first image data to the image data of the second item.

[0172] In one embodiment of the present disclosure, the evaluation result includes information indicating the probability of successfully recognizing the second item in the second image data in the image recognition, and the output circuit outputs a signal for displaying a candidate setting value for the first attribute information that would result in the probability being greater than the threshold, if the probability is less than or equal to a threshold.

[0173] In one embodiment of the present disclosure, the parameters for determining the candidate setting values ​​are predetermined or selected by the user.

[0174] In one embodiment of the present disclosure, the generation circuit generates new second image data by further adjusting the second image data with at least one of the second attribute pieces of information relating to a feature quantity different from the lighting environment, and the output circuit outputs a signal for displaying the evaluation result for the new second image data.

[0175] In one embodiment of the present disclosure, the output circuit outputs a signal for displaying a list of evaluation results corresponding to each of the multiple new second image data sets, which are generated by performing different adjustments on each of the second attribute information sets.

[0176] In one embodiment of the present disclosure, the output circuit outputs a signal for displaying information prompting the user to decide whether or not to perform retraining using the plurality of second image data after the changes have been made in the list display.

[0177] In one embodiment of the present disclosure, the generation circuit generates the second image data using a Generative Adversarial Network (GAN).

[0178] In an information processing method according to one embodiment of the present disclosure, the information processing device generates second image data that reproduces an image in which a second item different from the first item is placed in the first environment, based on attribute information of first image data of a first item photographed in a first environment, evaluates the accuracy of image recognition for the second item based on the second image data, and outputs the evaluation result of the accuracy.

[0179] A program according to one embodiment of the present disclosure causes a computer to perform the following processes: generate a second image data that reproduces an image in which a second item different from the first item is placed in the first environment, based on attribute information of a first image data of a first item photographed in a first environment; evaluate the accuracy of image recognition for the second item based on the second image data; and output the result of the accuracy evaluation. [Industrial applicability]

[0180] One embodiment of this disclosure is useful for a product recognition system. [Explanation of symbols]

[0181] 1. Product Recognition System 11 Image acquisition unit 12 Product recognition department 13 Storage section 14 Misrecognition analysis section 15 Result display area 141 Image database for performance evaluation 142 Image Encoding Unit 143 Attribute operation section 144 Image generation unit 145 Evaluation Department

Claims

1. a generation circuit that generates second image data that reproduces an image of a second item, different from the first item, placed in a first environment based on attribute information of first image data obtained by photographing the first item in the first environment; an evaluation circuit that evaluates the accuracy of image recognition for the second article based on the second image data; an output circuit that outputs the accuracy evaluation result; An information processing device comprising:

2. the evaluation result includes information indicating a probability that the image recognition will succeed in recognizing the second article in the second image data, the output circuit outputs a signal for displaying information indicating a probability of successful recognition of the second item in the second image data. The information processing device according to claim 1 .

3. the output circuit outputs a signal for distinguishing and displaying information about the second article for which the probability of successful recognition of the second article in the second image data exceeds a predetermined threshold value from information about the second article for which the probability does not exceed the predetermined threshold value. The information processing device according to claim 2 .

4. the generation circuit generates the second image data by changing attribute information of image data of the second item based on attribute information of the first image data. The information processing device according to claim 1 .

5. the generation circuit generates the second image data by applying first attribute information relating to a lighting environment for the image recognition, among attribute information of the first image data, to image data of the second item. The information processing device according to claim 4 .

6. the evaluation result includes information indicating a probability that the image recognition will succeed in recognizing the second article in the second image data, when the probability is equal to or less than a threshold, the output circuit outputs a signal for displaying candidates for the setting value of the first attribute information whose probability is greater than the threshold. The information processing device according to claim 5 .

7. The parameters that determine the setting value candidates are predetermined or selected by a user. The information processing device according to claim 6 .

8. the generation circuit further adjusts at least one of second attribute information relating to a feature amount different from the lighting environment for the second image data to generate new second image data; the output circuit outputs a signal for displaying the evaluation result for the new second image data. The information processing device according to claim 5 .

9. the output circuit outputs a signal for displaying a list of the evaluation results corresponding to each of the plurality of new second image data generated by performing different adjustments on the second attribute information. The information processing device according to claim 8 .

10. the output circuit outputs a signal for displaying, in the list display, a prompt to input information for determining whether or not to perform re-learning using the plurality of second image data after the change. The information processing device according to claim 9 .

11. The first image data is image data determined to be erroneously recognized in the image recognition. The information processing device according to claim 1 .

12. The generation circuit generates the second image data using a generative adversarial network (GAN). The information processing device according to claim 1 .

13. The information processing device generating second image data that reproduces an image of a second item different from the first item placed in the first environment based on attribute information of first image data obtained by photographing the first item in the first environment; Evaluating the accuracy of image recognition for the second item based on the second image data; outputting the evaluation result of the accuracy; Information processing methods.

14. On the computer, a process of generating second image data that reproduces an image of a second item, different from the first item, placed in a first environment based on attribute information of first image data obtained by photographing the first item in the first environment; A process of evaluating the accuracy of image recognition for the second item based on the second image data; a process of outputting the accuracy evaluation result; A program that executes the following.

Images