String extraction device, string extraction method, and string extraction program

Abstract

To provide a technology for easily extracting a character string displayed on a commodity, or the like.SOLUTION: A character string extraction apparatus 10 includes: a reception part 101 which receives a format of a character string containing one or more types of characters; a storage part 102 which stores the format; an acquisition part 103 which acquires an image containing a character string of an extraction object; a processing part 104 which recognizes a character from the image, and an extraction part 106 which extracts a character string corresponding to the format from the characters recognized from the image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a character string extraction device, a character string extraction method, and a character string extraction program. [Background technology]

[0002] In logistics warehouses, stores, factories, etc., inventory management of products is carried out by reading barcodes, expiration dates, and product control numbers when inspecting incoming and outgoing products. Patent documents 1 to 3 are known as prior art technologies for reading characters displayed on products, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6938952 [Patent Document 2] Patent No. 6878739 [Patent Document 3] Patent No. 3339122 Summary of the Invention [Problem to be solved by the invention]

[0004] Barcodes are read using a dedicated terminal that uses laser light, but expiration dates and product control numbers have to be visually checked and then manually entered by the product manager, which is a time-consuming process.

[0005] The present invention has been made in view of the above circumstances, and has an object to provide a technique that enables easy extraction of character strings displayed on products and the like. [Means for solving the problem]

[0006] A character string extraction device according to one embodiment of the present invention includes a receiving unit that receives a format of a character string including one or more types of characters, a storage unit that stores the format, an acquisition unit that acquires an image including the character string to be extracted, a processing unit that recognizes characters from the image, and an extraction unit that extracts a character string that conforms to the format from the characters recognized from the image.

[0007] A character string extraction method according to one aspect of the present invention is a character string extraction method performed by a character string extraction device, which receives a format for a character string including one or more types of characters, stores the format, acquires an image including the character string to be extracted, recognizes characters from the image, and extracts a character string that conforms to the format from the characters recognized from the image.

[0008] A character string extraction program according to one aspect of the present invention causes a computer to function as the character string extraction device. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a technique that can easily extract character strings displayed on products and the like. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 is a diagram illustrating a configuration of a terminal device. [Figure 2] FIG. 10 is a diagram showing a processing flow for setting the format of a character string to be extracted. [Figure 3] FIG. 10 is a diagram illustrating an example of a setting screen. [Figure 4] FIG. 10 is a diagram showing a processing flow for extracting character strings to be extracted. [Figure 5] FIG. 10 is a diagram illustrating an example of an execution screen. [Figure 6] FIG. 10 is a diagram illustrating an example of a list display screen. [Figure 7] FIG. 1 illustrates an application example of a character string extraction device. [Figure 8] FIG. 10 is a diagram illustrating an example of a character string extraction result. [Figure 9]This is an example of application of a character string extraction device. [Figure 10] FIG. 1 illustrates an application example of a character string extraction device. [Figure 11] FIG. 1 illustrates an application example of a character string extraction device. [Figure 12] FIG. 1 illustrates an application example of a character string extraction device. [Figure 13] FIG. 10 is a diagram illustrating an example of morphological transformation. [Figure 14] FIG. 10 is a diagram illustrating an example of the result of applying morphological transformation. [Figure 15] 10A and 10B are diagrams illustrating an example of extraction of character strings based on relative positional relationships between character strings. [Figure 16] 10A and 10B are diagrams illustrating an example of extraction of character strings based on relative positional relationships between character strings. [Figure 17] 10A and 10B are diagrams illustrating an example of extraction of character strings based on relative positional relationships between character strings. [Figure 18] FIG. 2 is a diagram illustrating a hardware configuration of the character string extraction device. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0012] [Summary of the Invention] This invention discloses a technology for extracting expiration dates, product control numbers, etc. from characters analyzed using OCR technology. Specifically, character strings such as expiration dates and product control numbers that were previously entered manually are automatically extracted from images. In particular, since there is no set format for product control numbers, a setting screen is designed to allow users to customize the format.

[0013] Note that characters include Roman letters, Greek letters, special characters, uppercase letters, lowercase letters, half-width characters, full-width characters, etc., which are commonly called the alphabet. Characters also include numbers and symbols. Numbers include Roman numerals and Arabic numerals. Symbols include hyphens, commas, parentheses, colons, etc. A string is a continuous sequence of any combination of letters, numbers, and symbols.

[0014] Hereinafter, in this embodiment, a character string extraction device that extracts character strings will be described as an example. In particular, the character string extraction device will be described as being implemented in a terminal device such as a smartphone terminal or a tablet terminal so that users can easily handle it at product receiving / shipping inspection locations.

[0015] [Terminal device configuration] 1 is a diagram showing the configuration of a terminal device 1 according to this embodiment. The terminal device 1 includes a character string extraction device 10 that extracts a character string to be extracted from an image, an imaging device 20 that captures the character string to be extracted, and a display device 30 that displays the extracted character string. The imaging device 20 is a camera included in the terminal device 1. The display device 30 is a display included in the terminal device 1.

[0016] Next, we will explain the functions of the character string extraction device 10. As shown in Figure 1, the character string extraction device 10 includes a reception unit 101, a storage unit 102, an acquisition unit 103, a processing unit 104, a determination unit 105, and an extraction unit 106.

[0017] The receiving unit 101 has a function of receiving the format of the character string to be extracted and storing the format in the storage unit 102. Specifically, the receiving unit 101 has a function of outputting to the display device 30 a setting screen for setting the format of the character string to be extracted, which allows one or more types of characters, the number of consecutive characters of each type, and the order of the character types to be set arbitrarily, and allowing the user to set the desired format of the character string.

[0018] The reception unit 101 also has a function of outputting an execution screen for executing extraction of character strings to be extracted to the display device 30. The execution screen includes operation buttons provided for each character string to be extracted. When a user presses a desired operation button, extraction of the desired character string corresponding to the pressed operation button begins.

[0019] The storage unit 102 has a function of storing the format of the character string to be extracted that is accepted by the accepting unit 101. The storage unit 102 also has a function of storing the character string extracted by the extracting unit 106. The storage unit 102 can store various data handled by the character string extraction device 10 and the terminal device 1.

[0020] The acquisition unit 103 has a function of acquiring an image including a character string to be extracted from the imaging device 20. In particular, the acquisition unit 103 has a function of repeatedly acquiring images including a character string to be extracted until a character string that matches the format of the character string to be extracted set by the user is extracted. Note that the acquisition unit 103 may be the imaging device 20 itself.

[0021] The processing unit 104 has a function of recognizing characters by OCR processing from the image acquired by the acquisition unit 103. The processing unit 104 also has a function of multiplying each pixel value of the image by a certain coefficient before recognizing characters by OCR processing.

[0022] The determining unit 105 has a function of reading the format of the character string to be extracted from the storage unit 102 and determining whether or not the characters recognized from the image by the processing unit 104 include a character string that conforms to the format.

[0023] The extraction unit 106 has a function of extracting a character string that conforms to the above format from among the characters recognized from the image by the processing unit 104. The extraction unit 106 also has a function of outputting the extracted character string to the display device 30 and storing the character string in the storage unit 102.

[0024] [Operation of the string extraction device] Next, the operation of the character string extraction device 10 will be described.

[0025] First, a method for setting the format of a character string to be extracted will be described with reference to Fig. 2, which shows the processing flow for setting the format of a character string to be extracted.

[0026] First, the receiving unit 101 of the character string extraction device 10 outputs a setting screen for setting the format of the character string to be extracted to the display device 30 (step S101).

[0027] Next, after the user inputs the format of the character string to be extracted on the setting screen, the receiving unit 101 of the character string extraction device 10 registers the input format of the character string to be extracted in the storage unit 102 (step S102).

[0028] An example of the setting screen is shown in Fig. 3. The setting screen 40 has a type selection field 41 for selecting the type (type) of character, a consecutive number selection field 42 for selecting the consecutive number of selected characters, and a format display field 43 for displaying the format of the character string to be set.

[0029] One or more inverted triangle buttons 411 are pre-displayed in the type selection field 41. When the user touches the button 411 with their finger, a list is displayed. The list contains each character type (more precisely, representative examples of each type), such as uppercase Roman letters, lowercase Roman letters, numbers, and hyphens. When a desired type is selected from the list, the character type is determined and displayed in the format display field 43.

[0030] The number of character types (number of types) can be increased or decreased. When two or more buttons 411 are touched to select two or more character types, the character types in the format are determined in order from the top row on the setting screen 40. In the example of FIG. 3, the order is determined as follows: capital Roman letters in the first row, numbers in the second row, and capital Roman letters in the third row. In FIG. 3, "A" represents a typical example of a capital Roman letter type, and "1" represents a typical example of a number type.

[0031] In the consecutive number selection field 42, for the character type selected in the type selection field 41, a consecutive number selection bar 421 for selecting the consecutive number of that character type is displayed with its length variable, and the Roman numerals for the consecutive number are written on the left side.

[0032] For example, if the user touches the right end of the consecutive number selection bar 421 with his / her finger and slides it to the right to the desired consecutive number, the consecutive number of the character type selected in the type selection field 41 is determined and displayed in the format display field 43.

[0033] 3, the number of consecutive uppercase Roman letters in the first row is set to 4, the number of consecutive numbers in the second row is set to 2, and the number of consecutive uppercase Roman letters in the third row is set to 1. As a result, "AAAA11A" is displayed in the format display field 43, and this "AAAA11A" is registered in the storage unit 102 as the format of the character string to be extracted.

[0034] So far, we have explained how to set the format of the string to be extracted. Note that multiple formats can be set for the string to be extracted. When inspecting the product upon arrival or shipment, the user checks the product control number printed on the surface of the product box and registers the format of the string related to that product control number. For best-before dates, formats for best-before dates are pre-set, such as "2022.09.28", "22.09.28", "22.9.28", "220928", "2022.09", and "22.09". Of course, you can also register the format of the string related to the best-before date.

[0035] Next, a method for extracting character strings to be extracted will be described. Fig. 4 is a diagram showing a processing flow for extracting character strings to be extracted. It is assumed that an execution screen for extracting character strings to be extracted is displayed on the display device 30 of the terminal device 1.

[0036] First, the user turns on the imaging device 20 of the terminal device 1 and captures an image of the barcode printed on the surface of the product box (step S201).

[0037] Next, the acquisition unit 103 of the character string extraction device 10 acquires an image including a barcode captured by the imaging device 20 (step S202).

[0038] Next, the processing unit 104 of the character string extraction device 10 acquires data information in the barcode from the acquired image including the barcode, and the extraction unit 106 outputs the data information to the display device 30 and registers it in the storage unit 102 (step S203).

[0039] So far, we have explained how to extract barcode information. As explained in the problem description above, barcodes are read using a dedicated terminal that uses laser light, but here we have explained how to read them from an image. From here on, we will explain how to extract character strings such as expiration dates and product control numbers from an image.

[0040] Next, when the user captures an image of the expiration date printed on the surface of the product box, the acquisition unit 103 of the character string extraction device 10 acquires the image including the expiration date captured by the imaging device 20 (step S204).

[0041] Next, the user touches the operation button for extracting the expiration date displayed on the execution screen with their finger and keeps touching it (step S205), which starts the process of extracting the character string of the expiration date.

[0042] Next, the processing unit 104 of the character string extraction device 10 multiplies each pixel value of the acquired image by a certain coefficient, and then recognizes characters in the image by OCR processing (step S206).

[0043] In this way, by multiplying each pixel value of an image by a certain coefficient, the shading of each pixel value becomes clearer, improving the accuracy of character recognition in subsequent OCR processing. For example, characters can be recognized with high accuracy even in dark places, when characters are faded, or when product packaging is wrinkled. Note that OCR (Optical Character Recognition) is an example of a character recognition method. Other character recognition methods may also be used.

[0044] Next, the determination unit 105 of the character string extraction device 10 reads the format of the character string related to the expiration date from the storage unit 102, and determines whether or not the characters recognized by the processing unit 104 include a character string that conforms to the format (step S207).

[0045] If there is no character string that matches the format, the process returns to step S204. On the other hand, if there is a character string that matches the format, the extraction unit 106 of the character string extraction device 10 extracts the character string from the characters recognized by the processing unit 104, and outputs the character string to the display device 30 (step S208).

[0046] After that, the acquisition unit 103 of the character string extraction device 10 stops the acquisition process of the image including the expiration date for a certain period of time (step S209).

[0047] Then, when the user refers to the character string displayed on the display device 30 and releases his / her finger from the operation button for extracting the expiration date because the expiration date has been correctly extracted (step S210), the extraction unit 106 of the character string extraction device 10 registers the extracted character string in the storage unit 102 (step S211). This ends the process of extracting the character string of the expiration date.

[0048] Here, a case where there is no character string that conforms to the format as a result of the determination made in step S207 will be described. The acquisition unit 103 of the character string extraction device 10 repeatedly executes the process of acquiring 30 images per second, and in this case, repeatedly acquires images including the expiration date (acquires a video including the expiration date) until a character string that conforms to the format is extracted.

[0049] That is, while the operation button for extracting the expiration date is continuously touched, the character string extraction device 10 repeats the image acquisition process by the acquisition unit 103, the character recognition process by the processing unit 104, and the determination process of the presence or absence of a character string by the determination unit 105 until a character string that matches the format is extracted. This makes it possible to reliably extract the character string to be extracted.

[0050] However, there is a possibility that the extracted character string may not be the one intended by the user. For example, if the expiration date and best-before date are both written, the character string format is similar, so there is a possibility that the character string for the best-before date will be extracted instead of the best-before date.

[0051] Therefore, even if a character string that conforms to the format is extracted, as long as the operation button for extracting the expiration date continues to be touched, the character string extraction device 10 repeats the image acquisition process by the acquisition unit 103, the character recognition process by the processing unit 104, and the process of determining whether or not a character string is present by the determination unit 105. Then, when the user releases his / her finger from the operation button when the intended character string is extracted, the character string extraction device 10 registers the character string displayed on the display device 30 at that time.

[0052] Up to this point, the case of extracting an expiration date has been described as an example. The same applies to the case of extracting a product control number. When a user touches with a finger the operation button for the product control number displayed on the execution screen and maintains the touch, the character string extraction device 10 executes steps S204 to S211 for the character string of the product control number.

[0053] An example of the execution screen is shown in Fig. 5. The execution screen 50 includes an image display field 51 for displaying an image of the product P captured by the imaging device 20, and an extraction process display field 52 for displaying the character string extraction results and operation buttons.

[0054] 5(a) is a diagram showing how barcode information of a product P is extracted. When the imaging process is started and the user positions the barcode of the product P at the center of the red auxiliary line 511 provided in the image display field 51, the character string extraction device 10 acquires data information in the barcode from the image including the barcode of the product P and displays the data information in the barcode display field 521 in the extraction process display field 52.

[0055] 5(b) is a diagram showing how to extract the expiration date of product P. When the imaging process is started and the user continues to touch the operation button B1 for the expiration date in the extraction process display field 52 and positions the expiration date of product P at the center of the auxiliary line 511, the character string extraction device 10 acquires the character string of the expiration date from the image including the expiration date of product P and displays the character string of the expiration date in the expiration date display field 522 in the extraction process display field 52.

[0056] 5(c) is a diagram showing how the product control number of product P is extracted. After starting the imaging process, the user continues to touch the operation button B2 for the product control number in the extraction process display field 52 and positions the product control number of product P at the center of the auxiliary line 511. The character string extraction device 10 then acquires the character string of the product control number from the image including the product control number of product P and displays the character string of the product control number in the product control number display field 523 in the extraction process display field 52.

[0057] Thereafter, when the user touches the registration button B3 in the extraction processing display field 52, the character string extraction device 10 registers the data information, expiration date, and product control number in the barcode displayed in the extraction processing display field 52 in association with each other.

[0058] Furthermore, when the user touches the List button B4 in the extraction process display field 52, the character string extraction device 10 displays a list display screen 60 as exemplified in Fig. 6. The list display screen 60 displays the expiration date and the like in the barcode of each product that has been registered in the past. When the user touches the Return to Photography button B5 in the list display screen 60, the character string extraction device 10 displays the execution screen 50 shown in Fig. 5.

[0059] [Variations] The present invention can be applied to extracting various arbitrary character strings in addition to expiration dates and product control numbers. For example, by registering the format "111-1111-1111" or "111-1111," it is possible to extract character strings such as telephone numbers and postal codes.

[0060] [Effects of the embodiment] According to this embodiment, the character string extraction device 10 includes a receiving unit 101 that receives a format of a character string including one or more types of characters, a storage unit 102 that stores the format, an acquiring unit 103 that acquires an image including the character string to be extracted, a processing unit 104 that recognizes characters from the image, and an extracting unit 106 that extracts a character string that fits the format from the characters recognized from the image, thereby making it possible to easily extract a character string displayed on a product or the like.

[0061] This allows specific character strings, such as expiration dates and product control numbers, to be extracted without manual input in situations where they must be extracted, for example, in a logistics warehouse, store, factory, etc. This reduces the amount of user operation required on the terminal device 1. As a result, it is possible to reduce the time required to inspect products, which also leads to the effect of reducing labor costs.

[0062] Furthermore, according to this embodiment, the acquisition unit 103 repeatedly acquires images including the character string to be extracted until a character string that conforms to the format is extracted, so that the character string to be extracted can be extracted reliably.

[0063] Furthermore, according to this embodiment, the receiving unit 101 outputs a format setting screen including the type of character and the number of consecutive characters of that type to the display device 30, so that the format of the character string to be extracted can be easily customized.

[0064] Furthermore, according to this embodiment, the reception unit 101 outputs to the display device a character string extraction execution screen including an operation button provided for each character string to be extracted, and when the operation button is pressed, extraction of the character string corresponding to the pressed operation button is started, so that the character string to be extracted can be easily extracted.

[0065] Furthermore, according to this embodiment, the acquisition unit 103, the processing unit 104, and the determination unit 105 repeat their respective processes while the operation button is pressed until a character string that conforms to the format is extracted, thereby ensuring that the character string to be extracted can be extracted reliably.

[0066] Furthermore, according to this embodiment, even if a character string that conforms to the format is extracted, the acquisition unit 103, processing unit 104, and judgment unit 105 repeat their respective processes while the operation button is pressed, so that the character string intended by the user can be extracted.

[0067] Furthermore, according to this embodiment, the processing unit 104 multiplies each pixel value of the image containing the character string to be extracted by a certain coefficient, thereby improving the accuracy of character recognition. Characters can be recognized with high accuracy even in dark places, when the characters are faded, or when the product packaging is wrinkled.

[0068] [Variations] A plurality of character strings to be extracted may be extracted at once.

[0069] First, the acquisition unit 103 acquires an image including both the expiration date and the product control number.

[0070] Next, when the user touches an operation button displayed on the execution screen with a finger, the processing unit 104 recognizes all characters in the image by OCR processing.

[0071] Next, the determination unit 105 reads out the format of each character string related to the expiration date and the product management number from the memory unit 102, and determines at once whether or not there is a character string among the characters recognized by the processing unit 104 that matches the format of each of the read character strings.

[0072] If there are character strings that match each format, the extraction unit 106 extracts the corresponding character strings from the characters recognized by the processing unit 104 and outputs them to the display device 30 all at once on the same screen.

[0073] In this way, it is determined at once whether or not there is a character string that matches each of the formats of a plurality of character strings, so that a plurality of character strings to be extracted can be extracted at once.

[0074] [Application example] An application example of the character string extraction device 10 will be described.

[0075] (Application Example 1) The character string extraction device 10 can be applied to receipts. For example, it can be applied to receipts of parking lots, toll roads, department stores, supermarkets, convenience stores, etc. However, Japanese receipts also include hiragana, etc. Also, the formats of receipts are not uniform.

[0076] Therefore, the reception unit 101 has a function of receiving Japanese characters such as hiragana, full-width katakana, half-width katakana, and Chinese characters, and receiving the format of the character string to be extracted in such a way that Japanese characters can also be combined.

[0077] Also, the reception unit 101 has a function of receiving the format of the character string to be extracted for each type of receipt.

[0078] The operation of the character string extraction device 10 in Application Example 1 will be described.

[0079] First, the user inputs in advance, for each receipt, the name of the item to be extracted and the format of the character string to be extracted corresponding to the item to be extracted. At this time, the reception unit 101 associates the name of the item to be extracted and the format of the character string to be extracted input by the user with each other, and further associates them with the receipt format ID input by the user, and registers them in the storage unit 102.

[0080] For example, the name of the item to be extracted (such as parking lot name, settlement No, receipt date, receipt time, delivery date, delivery time, parking fee, etc.) and the format of the character string to be extracted (such as 「AAA 亜亜亜」 corresponding to “ABC Parking Lot”) are registered in association with the receipt format ID for the ABC parking lot. 「亜」 is a representative example of the Chinese character type. The receipt format for parking lot XYZ is registered in the same way.

[0081] Next, the user photographs the receipt for ABC parking lot with the terminal device 1 (imaging device 20 in FIG. 1) as shown in FIG. 7. The user also selects the receipt format for ABC parking lot from two types of receipt formats on the terminal device 1 (display device 30 in FIG. 1). At this time, the reception unit 101 obtains the receipt format for ABC parking lot selected by the user from the storage unit 102.

[0082] Next, the processing unit 104 recognizes characters from within the image of the receipt for the ABC parking lot by OCR processing.

[0083] Next, the determination unit 105 determines whether or not the characters recognized by the processing unit 104 include character strings that respectively conform to the character string formats related to the receipt format for the ABC parking lot.

[0084] If there is a character string that matches each format, the extraction unit 106 extracts the character string from the characters recognized by the processing unit 104, associates it with the corresponding item to be extracted, and outputs it to the display device 30.

[0085] Of course, it is also possible to extract character strings from receipts for Parking Lot XYZ by using the receipt format for Parking Lot XYZ. Figure 8 shows the results of extracting character strings to be extracted from receipts for Parking Lot ABC and Parking Lot XYZ.

[0086] In this way, since Application Example 1 also uses Japanese characters, the character string extraction device 10 can be applied to Japanese receipts. Furthermore, since the format of the character string to be extracted is registered for each type of receipt, it is possible to eliminate the need to change the format of the character string for each different type of receipt, and it is possible to more easily extract the character string to be extracted.

[0087] (Application example 2) The character string extraction device 10 can be applied to a container.

[0088] As shown in FIG. 9, the character string extraction device 10 can extract the container number, container size, container type, weight, etc. painted on the surface of the rear door by capturing an image of the vicinity of the information display area AR1 of the rear door of the container.

[0089] Furthermore, the character string extraction device 10 can extract the character string from the seal sticker that is affixed to or covers the fastening key by capturing an image of the area AR2 of the fastening key on the rear door of the container.

[0090] In addition, the character string extraction device 10 can extract character strings of vehicle numbers of container transport vehicles, etc. The character string extraction device 10 can also extract character strings displayed on vehicles, ships, aircraft, etc. For example, it can also extract character strings such as loading information displayed on the side or back of a mixer truck.

[0091] (Application example 3) The character string extraction device 10 can be applied to case marks on luggage.

[0092] 10, the character string extraction device 10 can extract character strings printed on the side of a case mark, etc. The character string extraction device 10 can store the extracted character strings and the image of the case mark in association with each other, and can also convert this information into a QR code and output it to a label printer.

[0093] (Application example 4) The character string extraction device 10 can be applied to slips.

[0094] As shown in FIG. 11, the character string extraction device 10 can extract character strings written on delivery slips. The character string extraction device 10 can also extract handwritten character strings. The character string extraction device 10 can also automatically correct typos in addresses caused by typos or skipped characters using natural language processing technology. For example, the character string extraction device 10 changes a misspelling of "Shinagawa Ward, Tokyo" to "Shinagawa Ward, Tokyo." It adds "Tokyo" immediately before "Shinagawa Ward."

[0095] (Application example 5) The character string extraction device 10 can be applied to labels.

[0096] As shown in Fig. 12, the character string extraction device 10 can extract character strings from labels attached to cardboard boxes or folding containers. For example, character strings and data information in barcodes are extracted from labels including barcodes as shown in Fig. 12(a). Character strings are extracted from labels not including barcodes as shown in Fig. 12(b).

[0097] [Image processing method before OCR processing] In step S206, a method has been described in which each pixel value of an image is multiplied by a certain coefficient before OCR processing to clarify the shading of each pixel value and improve the accuracy of character recognition in the subsequent OCR processing.

[0098] To achieve the same effect, the processing unit 104 also has a function to perform morphological transformation on each pixel of the image before OCR processing. Specifically, the processing unit 104 changes the pixel value of each pixel included in the image based on the brightness relationship with surrounding pixels, and recognizes characters from the image after the change.

[0099] Morphological transformation is a technique for adjusting the size of an object in an image by changing the pixel value of each pixel in the image while taking into account the pixel values ​​of neighboring pixels. For example, as shown in Figure 13, an expansion process is performed to increase the number of black pixels in the original image or the image after morphological transformation, and an erosion process is performed to decrease the number of black pixels.

[0100] Specifically, the difference in color between a given pixel and its neighboring pixels is determined, and when performing expansion processing, for example, if the given pixel is black and the neighboring pixels are white, the color of the neighboring pixels is changed from white to black. In other words, the binary pixel value of each pixel is changed taking into account the brightness relationship with the surrounding pixels.

[0101] 14 is a diagram showing an example of the result of applying morphological transformation. The processing unit 104 first performs expansion processing and then contraction processing on the original image acquired in step S204. Thereafter, the processing unit 104 recognizes characters in the image by OCR processing.

[0102] In this way, by performing the expansion process before the contraction process, it is possible to connect dot-printed or faintly printed characters into meaningful characters, and to make the entire character clearer. On the other hand, if the amount of ink used for printing is large, it is possible to perform only the contraction process.

[0103] The order in which the expansion and contraction processes are performed and the selection of either process are arbitrary. The pixel expansion and contraction rates can also be determined arbitrarily. They can be set by the user, or the same or different fixed rates can be used for each rate.

[0104] According to this image processing method, before OCR processing, the pixel value of each pixel in the image is changed based on the brightness relationship with surrounding pixels, thereby improving the accuracy of character recognition in subsequent OCR processing and enabling more accurate extraction of the target character string.

[0105] The image processing method using morphological transformation described here may be executed in combination with an image processing method that multiplies each pixel value by a constant coefficient.

[0106] [How to extract text after OCR processing] When extracting multiple character strings at once, if the formats of the multiple character strings to be extracted are the same, the formats of the multiple character strings will match one or more character strings, which may result in an extraction result that is not as intended.

[0107] Therefore, a method for extracting character strings that takes into consideration the relative coordinates between character strings will be described.

[0108] The extraction unit 106 has a function of extracting character strings based on the relative positional relationship of a plurality of character strings after the OCR process in step S206 and before the match determination of the character strings in step S207. A specific example of this extraction method will be described below.

[0109] (Method 1 for extracting strings based on relative positional relationships between strings) In extraction method 1, the character string to be extracted is extracted based on the relative positional relationship between the item to be extracted (=character string) and the character string to be extracted.

[0110] Therefore, the extraction unit 106 has a function of extracting character strings to be extracted based on the relative coordinates of the items (keywords) to be extracted.

[0111] The operation of the character string extraction device 10 in extraction method 1 will be described below. A receipt will be used as an example.

[0112] The user sets in advance the format of character strings that can be candidates for the amount, such as "number + yen" or "￥ + number." At this time, the receiving unit 101 registers the format of the character string to be extracted that is input by the user in the storage unit 102.

[0113] The user also sets in advance the name of the item to be extracted, such as "total amount." At this time, the receiving unit 101 registers the name of the item to be extracted input by the user in the storage unit 102.

[0114] Next, the user takes a photo of the receipt with the terminal device 1.

[0115] Next, the processing unit 104 recognizes characters from the image of the receipt by OCR processing.

[0116] Next, the extraction unit 106 extracts an item to be extracted (for example, "total amount") from the characters recognized by the processing unit 104, and determines the item to be extracted as a keyword (key information).

[0117] Next, as shown in FIG. 15, the extraction unit 106 surrounds each of the character strings M0 to M4 recognized by the processing unit 104 with a rectangle, and acquires the position coordinates of four points of each rectangle.

[0118] Thereafter, for the keyword character string M0, the extraction unit 106 uses the position coordinates of four points of the rectangle of the character string M0 to find a line L connecting the midpoint TL of the left side of the rectangle and the midpoint TR of the right side of the rectangle, and extracts a rectangular character string M2 having a left side SL that intersects with the line L. The other character strings M1, M3, and M4 are not extracted.

[0119] In this way, in extraction method 1, character strings are extracted based on the relative coordinates with respect to the keyword, so that character strings can be narrowed down uniquely by the keyword, and the desired character string can be accurately extracted.

[0120] Here, an example has been described in which the keyword is located next to the desired character string to be extracted, but the desired character string can also be extracted using a similar method if the keyword is located above or below the desired character string to be extracted.

[0121] Furthermore, if the keyword and the desired extraction target character string are positioned obliquely, the straight line L may be inclined at a predetermined angle, or a straight line connecting the upper right vertex and the lower left vertex of the keyword rectangle may be used.

[0122] (Method 2 for extracting strings based on relative positional relationships between strings) In extraction method 2, character strings to be extracted are extracted based on the relative positional relationship between the character strings to be extracted.

[0123] For example, there are cases where the character string to be extracted spans multiple lines. In the case of the container shown in Figure 9, the broad container number (DICU 12345 5 42G1) that includes the container number, container size, and container type is written across two lines. In this case, it is time-consuming to extract the character string for each line.

[0124] Therefore, the receiving unit 101 has a function of receiving the format of the character string to be extracted in association with the line position, and setting the combination of these as the extraction condition for the character string.

[0125] The determining unit 105 also has a function of determining whether or not the characters recognized by the processing unit 104 include a character string that meets the character string extraction conditions.

[0126] The operation of the character string extraction device 10 in extraction method 2 will be described below. The container shown in FIG. 9 is used as an example.

[0127] 16, the user sets "first line: AAAA 111111 1" and "second line: 11A1" as the format of the character string to be extracted. In other words, the user associates the line position with the format of the character string to be extracted. At this time, the receiving unit 101 registers these in the storage unit 102 as extraction conditions for the character string.

[0128] Next, the user takes a photo of the rear door of the container using the terminal device 1.

[0129] Next, the processing unit 104 recognizes characters from the image of the container rear door by OCR processing.

[0130] Next, the determination unit 105 determines whether or not there is a string of characters among the characters recognized by the processing unit 104 that meets the extraction criteria for a string in which the format of the string in the first line is "AAAA 111111 1" and the format of the string in the second line is "11A1."

[0131] If there is a character string that meets the character string extraction condition, extraction unit 106 extracts the character string and outputs it to display device 30. At this time, extraction unit 106 may concatenate the character strings in each extracted line and display them on one line. On the other hand, if there is no character string that meets the character string extraction condition, extraction unit 106 outputs an error message to display device 30.

[0132] In this way, extraction method 2 extracts character strings to be extracted based on the positional relationship between the character strings to be extracted, so even if the character string to be extracted spans multiple lines, the character string can be easily extracted.

[0133] (Method 3 for extracting strings based on relative positional relationships between strings) In extraction method 3, character strings to be extracted are extracted based on the relative positional relationship between the items to be extracted (=character strings).

[0134] For example, in the case of the labels shown in FIG. 12, the position of the character string to be extracted generally differs for each type of label, so there is a possibility that the extraction result will not be as intended.

[0135] Therefore, the receiving unit 101 has a function for receiving the positional relationships between multiple items to be extracted. The method for registering the positional relationships is arbitrary. For example, a method of specifying the position of each item to be extracted by touching it with a finger on a screen for specifying the positional relationship, or a method of inputting the position coordinates of the items to be extracted as numerical values, can be considered.

[0136] The receiving unit 101 also has a function of receiving the positional relationships between a plurality of items to be extracted for each type of label.

[0137] The determining unit 105 also has a function of determining whether or not the positional relationship of each extraction target item recognized by the processing unit 104 matches the positional relationship of the registered extraction target items.

[0138] The operation of the character string extraction device 10 in extraction method 3 will be described below. The label shown in FIG. 12(a) will be used as an example.

[0139] The user inputs the positional relationships between the items to be extracted in advance for each label type. At this time, the receiving unit 101 associates the positional relationships between the items to be extracted input by the user with the label format ID and registers them in the storage unit 102.

[0140] For example, as shown in Fig. 17, the positions on the screen of each extraction target item (such as "Tokyo Center") designated by the user are registered for each of the labels LB1 and LB2. Specifically, the position coordinates of the extraction target items, the distance between the positions of the extraction target items, the relative angle between the positions of the extraction target items, etc. are registered.

[0141] Next, the user photographs the label LB1 with the terminal device 1 (the imaging device 20 in FIG. 1). The user also selects the label format of the label LB1 from two types of labels on the terminal device 1 (the display device 30 in FIG. 1). At this time, the reception unit 101 acquires the label format of the label LB1 selected by the user from the storage unit 102.

[0142] Next, the processing unit 104 recognizes a character string from the image of the label LB1 by OCR processing.

[0143] Next, as shown in Fig. 17, the determination unit 105 surrounds each of the character strings M1 to M4 recognized by the processing unit 104 with a rectangle and calculates the four center points CP1 to CP4 of each rectangle. Thereafter, the determination unit 105 determines whether the positional relationship between the center points CP1 to CP4 of the character strings M1 to M4 matches the positional relationship between the items to be extracted according to the label format of the label LB1. If the positional relationships are similar, it may be determined that they match.

[0144] If the two positional relationships match, the extraction unit 106 extracts character strings in character string formats corresponding to the character strings M1 to M4 recognized by the processing unit 104, and outputs the character strings to the display device 30. If the two positional relationships do not match, the extraction unit 106 outputs an error message to the display device 30.

[0145] If, for example, label LB1 is photographed but the label format of label LB2 is selected, the positional relationship between the items to be extracted related to the label format of label LB2 does not match the positional relationship between the character strings recognized from the image of label LB1, and an error message will be displayed.

[0146] In this way, in extraction method 3, the positional relationship of the items to be extracted is registered for each type of label format, and the character string to be extracted is extracted based on the positional relationship of the items to be extracted for the desired label format, so the desired character string can be extracted accurately.

[0147] Here, we have explained an example in which character strings to be extracted are extracted based on the relative positional relationships between the items to be extracted. However, since the items to be extracted and the character strings to be extracted are generally displayed as a pair of left and right horizontal pairs (or a pair of top and bottom vertical pairs), character strings to be extracted may also be extracted based on the relative positional relationships between the character strings to be extracted, rather than between the items to be extracted.

[0148] [others] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the present invention.

[0149] The character string extraction device 10 of the present embodiment described above can be realized, for example, by using a general-purpose computer system including a CPU 901, a memory 902, a storage 903, a communication device 904, an input device 905, and an output device 906, as shown in Fig. 17. The memory 902 and the storage 903 are storage devices. In the computer system, the CPU 901 executes a predetermined program loaded onto the memory 902, thereby realizing each function of the character string extraction device 10.

[0150] The character string extraction device 10 may be implemented by one computer. The character string extraction device 10 may be implemented by multiple computers. The character string extraction device 10 may be a virtual machine implemented on a computer. The program for the character string extraction device 10 can be stored in a computer-readable recording medium such as an HDD, SSD, USB memory, CD, or DVD. The computer-readable recording medium is, for example, a non-transitory recording medium. The program for the character string extraction device 10 can also be distributed via a communication network. [Explanation of symbols]

[0151] 1...Terminal device 10...Character string extraction device 20...imaging device 30…Display device 101…Reception 102...Storage section 103…Acquisition part 104... Processing section 105…Judgment section 106...Extraction part 901...CPU 902...Memory 903…Storage 904...Communication equipment 905...Input device 906...Output device

Claims

1. a reception unit that receives a format of a character string including one or more types of characters; a storage unit that stores the format; an acquisition unit that acquires an image including a character string to be extracted; a processing unit for recognizing characters from the image; an extraction unit that extracts a character string that conforms to the format from the characters recognized from the image, The reception unit A character string extraction device outputs to a display device a character string extraction execution screen including an operation button provided for each character string to be extracted, and when one of the operation buttons is pressed, extraction of the character string corresponding to the pressed operation button is started.

2. The acquisition unit The character string extraction device according to claim 1 , further comprising: repeatedly acquiring images including the character string to be extracted until a character string conforming to the format is extracted.

3. The reception unit 2. The character string extraction device according to claim 1, wherein a format setting screen including the type of character and the number of consecutive characters of said type is output to a display device.

4. The acquisition unit and the processing unit 2. The character string extraction device according to claim 1, wherein each process is repeated while the operation button is pressed until a character string that matches the format is extracted.

5. The acquisition unit and the processing unit 5. The character string extraction device according to claim 4, wherein even if a character string that conforms to the format is extracted, the respective processes are repeated as long as the operation button is being pressed.

6. The processing unit The character string extraction device according to claim 1 , wherein each pixel value of the image including the character string to be extracted is multiplied by a constant coefficient.

7. The extraction unit The character string extraction device according to claim 1 , wherein an expiration date or a product control number is extracted as the character string conforming to the format.

8. The characters are: The character string extraction device according to claim 1 , wherein the character string includes numbers or symbols.

9. The character string extraction device The character string extraction device according to claim 1, which is provided in a terminal device.

10. The processing unit 2. The character string extraction device according to claim 1, wherein the pixel value of each pixel included in the image is changed based on a brightness relationship with surrounding pixels, and characters are recognized from the changed image.

11. The extraction unit 2. The character string extraction device according to claim 1, wherein the character string is extracted based on the relative positional relationship of a plurality of character strings.

12. In a character string extraction method performed by a character string extraction device, accepting a string format including one or more types of characters; storing the format; acquiring an image including a character string to be extracted; Recognizing characters from the image; extracting a character string that conforms to the format from the characters recognized from the image; In the receiving step, A character string extraction method in which a character string extraction execution screen including an operation button provided for each character string to be extracted is output to a display device, and when one of the operation buttons is pressed, extraction of the character string corresponding to the pressed operation button is started.

13. A character string extraction program that causes a computer to function as the character string extraction device according to any one of claims 1 to 11.

Images