Information processing system, information processing method, and information processing program

Abstract

To accurately identify a product while reducing a learning cost.SOLUTION: An information processing system includes: a first calculation unit that calculates a first inter-keyword feature amount for each product, based on both a product name of each of a plurality of products and a predetermined keyword relating to each of the products; a second calculation unit that calculates a second inter-keyword feature amount of an image captured by imaging an object product, based on both a character string extracted by character recognition from the captured image of the object product and the keyword; and an identification unit that calculates similarity between the first inter-keyword feature amount and the second inter-keyword feature amount for each product and identifies the object product from the products for which the similarity satisfies a predetermined threshold condition.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, there is a technology related to character recognition for recognizing characters attached to an object shown in an image.

[0003] For example, there is a technology for reading characters on a label attached to the end face of a steel material or characters engraved thereon (see Patent Document 1). In this technology, a character recognition is performed from a plurality of images captured by a plurality of cameras by combining a camera with a large field of view and a small camera.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, character recognition from an image (mainly OCR: Optical Character Recognition) inevitably causes misrecognition due to the influence of the imaging environment or the like. Therefore, there is a problem in recognizing the product name from the product image simply by performing character recognition.

[0006] The present disclosure has been made in view of the above circumstances, and an object thereof is to provide an information processing system, an information processing method, and an information processing program that can suppress the learning cost and accurately identify a product.

Means for Solving the Problems

[0007] The information processing system disclosed herein includes: a first calculation unit that calculates a first keyword feature quantity, which is a feature quantity for each product, based on the product name of each of a plurality of products and a pre-set keyword associated with each of the products; a second calculation unit that calculates a second keyword feature quantity, which is a feature quantity for an image of the target product, based on a string extracted by character recognition from an image of the target product and the keyword; and an identification unit that calculates the similarity between the first keyword feature quantity and the second keyword feature quantity for each product, and identifies the target product from among the products whose similarity satisfies a predetermined threshold condition. [Effects of the Invention]

[0008] The information processing system, information processing method, and information processing program disclosed herein have the effect of enabling accurate product identification while reducing learning costs. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a schematic diagram illustrating the method of this embodiment. [Figure 2] Figure 2 is a block diagram showing the hardware configuration of the information processing system 100. [Figure 3] Figure 3 is a block diagram showing the configuration of the information processing system 100 in this embodiment. [Figure 4] Figure 4 shows an example of keywords for each product stored in the product information database. [Figure 5] Figure 5 shows an example of the similarity between keywords. [Figure 6] Figure 6 shows an example of the first keyword features for product A. [Figure 7] Figure 7 shows an example of the first keyword features calculated for each product. [Figure 8] Figure 8 shows an example of images taken from different sides of the product in question. [Figure 9] Figure 9 shows an example of the similarity score calculated for each product. [Figure 10] Figure 10 is a flowchart showing the flow of pre-processing in information processing by the information processing system 100. [Figure 11] Figure 11 is a flowchart showing the flow of recognition processing in information processing by the information processing system 100. [Modes for carrying out the invention]

[0010] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. In each drawing, identical or equivalent components and parts are given the same reference numerals. Furthermore, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from actual ratios.

[0011] An overview of the embodiments of this disclosure will be described. In this embodiment, a method is proposed for recognizing the product name by calculating feature quantities for keywords in registered product information and comparing them with the feature quantities of strings recognized from images of the captured product. In this embodiment, multiple images taken of multiple different sides of a product are used to extract strings from each side and compare the target products. The multiple images are assumed to be taken using a robot capable of capturing different sides of a product. Multiple robots may be used for capturing the images. Alternatively, a conveyor system may be used to capture different sides of the product. The products targeted by this embodiment are all products with strings attached, and packaged materials will be used as an example. Other products that can be applied include printed goods or drums, etc.

[0012] Figure 1 is a schematic diagram illustrating the method of this embodiment. Multiple images of the target product to be recognized are captured and input into the information processing system 100 of this embodiment. Multiple images are captured by multiple imaging devices (200) such as cameras. Keywords related to product information input by the user are registered in the information processing system. The information processing system identifies the product name of the target product by comparing the similarity of the feature quantities described below from the input of the multiple captured images and outputs it.

[0013] Figure 2 is a block diagram showing the hardware configuration of the information processing system 100.

[0014] As shown in Figure 2, the information processing system 100 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to be communicable with each other via a bus 19.

[0015] The CPU 11 is a central processing unit that executes various programs and controls each unit. That is, the CPU 11 reads a program from the ROM 12 or the storage 14 and executes the program using the RAM 13 as a work area. The CPU 11 performs control of each of the above components and various arithmetic processes according to a program stored in the ROM 12 or the storage 14. In the present embodiment, a prediction program is stored in the ROM 12 or the storage 14.

[0016] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores a program or data as a work area. The storage 14 is composed of a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs including an operating system and various data.

[0017] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0018] [[ID=二十一]]The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may adopt a touch panel method and function as the input unit 15.

[0019] The communication interface 17 is an interface for communicating with other devices such as terminals. For such communication, a wired communication standard such as Ethernet® or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi® may be used.

[0020] The functional configurations of the information processing system 100 will now be described. Figure 3 is a block diagram showing the configuration of the information processing system 100 in this embodiment. Each functional configuration is realized by the CPU 11 reading an information processing program stored in the ROM 12 or storage 14, loading it into the RAM 13, and executing it. The information processing system 100 is connected to the imaging device 200 via a network N. The imaging device 200 consists of multiple cameras or multiple robots equipped with cameras that image a target product, and transmits multiple images of different sides of the target product to the information processing system 100.

[0021] As shown in Figure 3, the information processing system 100 functionally consists of a product information database (DB) 102, a first calculation unit 110, a second calculation unit 112, and a specific unit 114.

[0022] Product Information DB102 stores the names of multiple products, along with keywords associated with each product. Keywords are pre-configured based on user input, drawing from product names, manufacturing company names, product descriptions, and symbols. Since the printed labels differ for each product, distinctive product strings are registered. Keywords should include strings and symbols useful for product identification.

[0023] Feature DB104 stores the first keyword features and the second inter-keyword features, which will be described later. In the following processing section, features will be read from Feature DB104 as needed and processed, so the explanation will be omitted.

[0024] Figure 4 shows an example of keywords for each product stored in the product information DB102. In Figure 4, an example is shown where multiple keywords related to a product are registered for each product name, but for the sake of explanation, arbitrary strings are given as product names and keywords. In the first line, the product name "AAAA" has the product name itself registered as keyword 1. In this way, the product name and keyword may be the same, but the product name used to identify the product and the product name used as a keyword are registered separately. In the second line, the product name "BBBB150" has the string "BBBB150" divided into keyword 1 "BBBB" and keyword 2 "150" and registered. Note that the string may be registered as keywords after being divided, or it may not be divided. Alternatively, the divided string and the undivided string may be registered as keywords separately. In the third line, the product "XXXXX" has the related keywords keyword 1 "yyyyy", "200", and "zzzzz" registered. In the third line of products, it is assumed that the product name is not included, but rather product information such as a description and quantity are attached as text.

[0025] The first calculation unit 110 calculates the first keyword feature quantity, which is a feature quantity for each product. The first calculation unit 110 stores the calculated first keyword feature quantity for each product in the feature quantity DB 104.

[0026] The processing of the first calculation unit 110 is divided into two stages: (1) First, the first calculation unit 110 obtains keywords for each product from the product information DB 102 and calculates the similarity between keywords for all combinations of keywords using each of the keywords included in all products.

[0027] Figure 5 shows an example of keyword similarity. In the matrix graph, each column and row of the matrix represents a set of keywords, and each cell in the row and column corresponds to the similarity between the keywords in the corresponding column and row. A similarity score closer to 0.0 indicates less similarity, while a score closer to -1.0 indicates greater similarity.

[0028] (2) Next, the first calculation unit 110 calculates, for each product, the similarity calculated for the keywords related to the product (product name) among the similarity between keywords, as the first keyword feature of the product. Figure 6 is an example of the first keyword feature of product A. In the example in Figure 6, for the sake of explanation, it is assumed that the keyword for product A is limited to one product name. This is the same case as the product in the first row of Figure 4 above. The similarity for a combination of one keyword and each keyword corresponds to one row of the keyword similarity in (A). Therefore, for product A, the keyword similarity in (B), i.e., the first keyword feature of the product, can be calculated by extracting one row of (A). Each of the similarities calculated for the keyword combination is an element of the array in the first keyword feature. Each row in (B) corresponds to a keyword. If a product has multiple keywords associated with it, the first keyword feature of the product can be calculated by extracting the row of the keyword combination for each keyword included in the product.

[0029] Figure 7 shows an example of the first keyword features calculated for each product. It can be seen that different first keyword features are calculated for each product.

[0030] The second calculation unit 112 calculates the second keyword inter-keyword feature, which is a feature of the image of the captured target product. The second keyword inter-keyword feature is calculated by making the similarity of each combination between each string extracted by character recognition from the image of the target product and each keyword into elements of an array. The second calculation unit 112 stores the calculated second keyword inter-keyword feature for the image of the target product in the feature database 104.

[0031] The keywords are all the keywords registered in the product information DB102. The images accepted by the second calculation unit 112 are each of several images. Each image is an image of multiple different sides of the target product taken from multiple different orientations. Since the product has different strings of characters on different sides, each side is imaged. The different sides should be those on which printing is customary on the product. For example, if the target is a material that is normally printed on the top and sides, then the top and sides should be the sides to be imaged. If the target is a product that is printed on the bottom, then the bottom should also be included. The different orientations for imaging should be one or more orientations that allow each of the different sides to be imaged. Images taken from different angles of the same side may also be included. The second calculation unit 112 performs character recognition on each image and extracts the strings of characters. The second calculation unit 112 calculates the second keyword similarity for each combination of the strings of characters extracted from the images and each of the keywords. In other words, in the processing of the second calculation unit 112, a second keyword feature corresponding to (B) in Figure 6 above is calculated for each string extracted from the image. In the following, when explaining matters common to the first keyword feature and the second keyword feature, we will simply refer to them as keyword features. Also, the "similarity between keywords" and the "similarity in the sequence of keyword features" mentioned above are the "individual similarities" when combined with the keywords. Therefore, the "individual similarities" are distinguished from the "similarity" of the identification unit 114 described later (the similarity when comparing each keyword feature for each product).

[0032] Figure 8 shows an example of images taken from different sides of the target product. Figure 8(A) is an image taken from the top, and (B) is an image taken from the side. Here, we will explain the necessity of using multiple images taken from different sides. The dotted line frame indicated by (rc) is the part where character recognition was performed. The product information on the top side of the product is as follows: (A) has the strings "XXXX" and "yyyyy200", and (B) has the string "zzzzz". Note that the strings attached to each side of the product are merely examples. As different keyword strings are attached to the product in this way, it is necessary to extract strings from multiple different sides in order to predict the product name.

[0033] The identification unit 114 calculates the similarity between the first keyword feature and the second keyword feature for each product, and identifies the target product from among the products whose similarity meets the threshold condition. The identification unit 114 determines that the threshold condition is met if any product exceeds the similarity threshold, and determines that the threshold condition is not met if no product exceeds the similarity threshold. The threshold can be set by the user to any value, and the threshold can be set heuristically or calculated statistically. In this embodiment, the threshold is set heuristically. In addition, different thresholds may be set for each product or product category.

[0034] Figure 9 shows an example of the similarity score calculated for each product. (A) is a graph when there are products that exceed the threshold, and (B) is a graph when there are no products that exceed the threshold. The threshold for the similarity score of each product is set between 2.0 and 3.0. In (A), the product shown in the bar graph at (p) is the product with the highest similarity score, and the identification unit 114 identifies this product that exceeds the threshold as the target product.

[0035] The identification unit 114 outputs the product name of the product corresponding to the similarity exceeding the threshold as the product name of the target product. In addition, as shown in (B) of Figure 9 above, the identification unit 114 outputs a flag for unregistered products if none of the products meet the threshold condition.

[0036] The similarity between features can be calculated using the Euclidean distance in equation (1) below. p and q are the features for each keyword in the keyword-to-keyword features (individual similarity), and n is the total number of keywords.

number

[0037] Furthermore, in calculating the similarity, the feature vector x' of the keyword is standardized for each keyword i, as shown in equation (2) below. i The standardized feature x' is obtained and used as p and q. i To calculate this, the feature x of keyword i i The mean (superscript x) and standard deviation (σ) are used. Note that the feature x i This represents each element of all features (individual similarities) included in the first keyword feature and the second keyword feature.

number

[0038] Furthermore, multiple thresholds for determining similarity may be set using different indicators. For example, in the calculation of Euclidean distance described above, a specific keyword showing a peak in similarity (distance) may be treated separately from other keywords, and a peak threshold may be set for the value at which the peak value of the specific keyword exceeds a predetermined value. Alternatively, a lower threshold may be set for the values ​​of other keywords other than the peak value. In this case, similarity is determined when all of the set thresholds are exceeded, and the threshold conditions are deemed to be met.

[0039] Here, we illustrate the advantages of using keyword feature similarity as in this embodiment. For example, suppose a string contains misrecognized characters. In this case, by using string similarity, it is possible to derive a similar string even if misrecognized characters are present. Also, suppose the user cannot visually determine which string is the product name. In this case, by calculating similarity for all strings, the product name can be identified regardless of where the string is written. Furthermore, suppose that the product name is recognized in segments during character recognition. In this case, by dividing the string, which is the unit for calculating similarity, into multiple short keywords, it is possible to handle cases where the product name is segmented.

[0040] If a flag is output for an unregistered product, the user will need to take action regarding that product. Upon receiving the flag for an unregistered product, the user refers to the keywords in the string extracted by character recognition or to an external product information database. If the user can identify the product information for the unregistered product based on the referral results, they register the keywords for that product in the product information DB 102. The information processing system 100 then recalculates the first keyword feature quantities for each product, including the keywords for the newly registered product.

[0041] (Process flow) Next, the operation of the information processing system 100 as an information processing method will be explained. Figures 10 and 11 are flowcharts showing the flow of information processing by the information processing system 100. Information processing is performed when the CPU 11 reads an information processing program from the ROM 12 or storage 14, loads it into the RAM 13, and executes it. The CPU 11 functions as a part of the information processing system 100 to execute the following processes. Depending on the process, necessary data is read from the product information DB 102 and feature quantity DB 104 and each process is executed.

[0042] The processing routine in Figure 10 is a pre-processing step in which the first keyword features for each product are calculated and registered in advance. The processing routine in Figure 11 is a recognition process that identifies the target product when an image of the target product is input. The information processing system 100 may also execute the processing routines in Figures 10 and 11 as a series of processes.

[0043] The preprocessing steps will now be explained. In step S100, the CPU 11 retrieves keywords for each product from the product information DB 102.

[0044] In step S102, the CPU 11 calculates the similarity between keywords for all combinations of keywords using each of the keywords included in all products.

[0045] In step S104, the CPU 11 calculates the first keyword feature for each product, using the similarity calculated for keywords related to the product (product name) among the similarity between keywords. This calculates the first keyword feature for each product.

[0046] In step S106, the CPU 11 registers the calculated first keyword features for each product in the feature database 104.

[0047] Next, the recognition process will be explained. In step S200, the CPU 11 acquires each of the multiple images of the target product captured from the imaging device 200.

[0048] In step S202, the CPU 11 performs character recognition on each image and extracts the corresponding text string.

[0049] In step S204, the CPU 11 calculates a second keyword similarity score for each combination of strings extracted from the image and each keyword.

[0050] In step S206, the CPU 11 calculates the similarity between the first keyword feature and the second keyword feature for each product. The similarity is calculated for each product.

[0051] In step S208, the CPU 11 determines whether there are any products whose similarity meets the threshold condition. If there are products that meet the threshold condition, the process proceeds to step S118; otherwise, the process proceeds to step S120.

[0052] In step S210, the CPU 11 outputs the product names of products whose similarity exceeds the threshold as the product name of the target product. If multiple products are applicable, the CPU 11 may output multiple product names as candidates, or it may output the product name with the highest similarity.

[0053] In step S212, the CPU 11 outputs a flag indicating an unregistered product.

[0054] As described above, the information processing system 100 according to this embodiment enables accurate product identification while reducing learning costs.

[0055] Furthermore, according to this embodiment, the only manual task required is registering product information, enabling a system that allows for easy product identification. Additionally, registering or deleting new products can be done simply by manipulating the registered product information to update the system. It is assumed that any OCR model can be used for character recognition, converting image sets into text strings, and this can be changed as needed. Furthermore, by using keyword features and feature similarity, product names can be identified regardless of the accuracy of the OCR model. Unregistered products can also be classified by setting a similarity threshold. Moreover, the learning cost can be reduced because there is no need for model training using large amounts of images or model updates.

[0056] It should be noted that the present invention is not limited to the embodiments described above, and various modifications and applications are possible without departing from the spirit of the invention.

[0057] For example, in the embodiments described above, the use of keywords or extracted strings as data for calculating inter-keyword features was explained as an example, but it is not limited to this. For example, if there is a distinctive color scheme or string position, such elements may be incorporated into the array of inter-keyword features.

[0058] Alternatively, the first calculation unit 110 may set keywords from strings extracted from each of a plurality of images of the product taken in advance, and calculate the first keyword feature quantities.

[0059] Furthermore, although the present specification describes an embodiment in which the program is pre-installed, it is also possible to provide the program stored on a computer-readable recording medium. [Explanation of symbols]

[0060] 100 Information Processing Systems 102 Product information DB 104 Feature Database 110 First Calculation Unit 112 Second Calculation Unit 114 Specific section

Claims

1. A first calculation unit calculates a first keyword feature quantity, which is a feature quantity for each product, based on the product name of each of the multiple products and a pre-set keyword associated with each of the said products. A second calculation unit calculates a second keyword feature quantity, which is a feature quantity of the image of the target product, based on a string extracted by character recognition from an image of the target product and the aforementioned keyword. A selection unit that calculates the similarity between the first keyword feature and the second keyword feature for each product, and identifies the target product from the products whose similarity satisfies a predetermined threshold condition, An information processing system that includes this.

2. The first calculation unit calculates the similarity between keywords for all combinations of keywords using each of the keywords included in all of the products, The information processing system according to claim 1, wherein for each of the aforementioned products, the similarity calculated for the keywords related to the product among the similarity between the keywords is calculated as the first keyword feature quantity of the said product.

3. The information processing system according to claim 1, wherein the keyword includes, for each product, at least one of the following: the product name, the name of the company that manufactures the product, a string of characters included in the description attached to the product, and a symbol attached to the product.

4. The aforementioned images are each of a plurality of images capturing multiple different surfaces of the product. The information processing system according to claim 1, wherein the second calculation unit extracts a string from each of the images.

5. The information processing system according to claim 1, wherein the first calculation unit sets the keyword from a string extracted from each of a plurality of images of the product taken in advance, and calculates the first keyword feature quantity.

6. The specified part is, When the similarity satisfies the threshold condition, the product name of the product whose similarity exceeds the threshold is output as the product name of the target product to identify the target product. The information processing system according to claim 1, which outputs a flag for an unregistered product if the similarity does not satisfy the threshold condition.

7. Based on the product names of multiple products and pre-defined keywords associated with each of the said products, the first keyword-specific feature quantity, which is a feature quantity for each product, is calculated. Based on the string extracted by character recognition from the image of the target product and the aforementioned keyword, a second keyword feature quantity, which is a feature quantity of the image of the target product, is calculated. The similarity between the first keyword feature and the second keyword feature for each product is calculated, and the target product is identified from the products whose similarity satisfies a predetermined threshold condition. An information processing method in which a computer performs the processing.

8. Based on the product names of multiple products and pre-defined keywords associated with each of the said products, the first keyword-specific feature quantity, which is a feature quantity for each product, is calculated. Based on the string extracted by character recognition from the image of the target product and the aforementioned keyword, a second keyword feature quantity, which is a feature quantity of the image of the target product, is calculated. The similarity between the first keyword feature and the second keyword feature for each product is calculated, and the target product is identified from the products whose similarity satisfies a predetermined threshold condition. An information processing program that instructs a computer to perform a task.

Images