Image processing device and image processing program

Abstract

To provide an image processing device and an image processing program that enhance the recognition precision for a letter string.SOLUTION: An image processing device 1 includes: an input interface 14 that obtains a plurality of images picked up by an imaging device; and a processor 11 which extracts a letter region letter by letter for each of the plurality of images, determines a first letter region contained in the one or more images among the plurality of images on the basis of the image matching on the plurality of images, identifies the one or more letter regions that constitute a letter string on the basis of the determination on the first letter region, and identifies the letter string on the basis of the identification of the one or more letter regions that constitute the letter string.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an image processing device and an image processing program. [Background technology]

[0002] Optical Character Recognition (OCR) processing is generally applied to document images captured by flatbed scanners. In recent years, there have been increasing opportunities to recognize characters in scenes captured by cameras or smartphones. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-201094 Summary of the Invention [Problem to be solved by the invention]

[0004] When trying to recognize characters from an image taken with a camera or smartphone, the image may have halation or shadows (blackouts) unless you take the image while taking note of the effects of ambient light. When halation or shadows occur in the image, there will be areas where the characters cannot be recognized.

[0005] The problem to be solved by the embodiments of the present invention is to provide a technique for improving the accuracy of character string recognition. [Means for solving the problem]

[0006] In one embodiment, an image processing device includes an interface and a processor. The interface acquires a plurality of images captured by an image capture device. The processor extracts a character region for each character in each of the plurality of images, determines a first character region included in one or more of the plurality of images based on image matching of the plurality of images, identifies one or more character regions constituting a character string based on the determination of the first character region, and identifies the character string based on the identification of the one or more character regions constituting the character string. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram illustrating an image processing apparatus according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of an object including a character string according to the embodiment. [Figure 3] FIG. 3 is a flowchart illustrating the procedure of image processing by the image processing device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an image captured by the imaging device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating the extraction process of a character region by the image processing apparatus according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating the process of determining identical character regions by the image processing apparatus according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating a process of specifying characters for each character region performed by the image processing apparatus according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment will be described with reference to the drawings.

[0009] (Configuration example) FIG. 1 is a block diagram illustrating an image processing device 1. As shown in FIG. The image processing device 1 is a device that processes multiple images of a character string captured by an imaging device 2. For example, the image processing device 1 is a device that identifies a character string of an object based on character recognition of multiple images. A character string is a sequence of one or more characters. A character is a concept that includes an identification code such as a number, symbol, code, or mark. A character string is sometimes called a subject. An object is an object that includes a character string. A character string may be directly printed on an object. An object may also have a sheet with a character string printed on it attached.

[0010] The image processing device 1 may be used in a warehouse, a store, or the like. When the image processing device 1 is used in a warehouse, the target object may be a cardboard box wrapped in a transparent sheet, or the like. The character string may be an expiration date, or the like. When the image processing device 1 is used in a store, the target object may be a menu in a clear folder, or the like. The character string may be a product name, or the like.

[0011] For example, the image processing device 1 may be, but is not limited to, a PC (Personal Computer), a tablet terminal, a smartphone, etc. The image processing device 1 may be any hardware having an image processing function. The image processing device 1 is an example of a character recognition device.

[0012] The photographing device 2 is a device that has a photographing function and acquires images based on photographing. The photographing device 2 may be a device that has a function of photographing moving images or a function of photographing still images. For example, the photographing device may be, but is not limited to, a camera, a tablet terminal with a camera, or a smartphone with a camera. The photographing device 2 may be any hardware that has a photographing function. The photographing device 2 photographs the same string of characters multiple times. For example, the photographing device 2 photographs the same string of characters from different viewpoints. Photographing the same string of characters from different viewpoints includes photographing the same string of characters from different directions. Photographing the same string of characters from different viewpoints includes photographing the same string of characters from different distances. The multiple images acquired based on photographing the string of characters multiple times by the photographing device 2 may be different images that show the string of characters. The multiple images include the string of characters.

[0013] The image processing device 1 includes a processor 11, a main memory 12, an auxiliary storage device 13, and an input interface 14. The components constituting the image processing device 1 are connected to each other so that signals can be input and output. In Fig. 1, the interface is indicated as "I / F."

[0014] The processor 11 corresponds to the central part of the computer of the image processing device 1. For example, the processor 11 is a CPU (Central Processing Unit), but is not limited to this. The processor 11 may be composed of various circuits. The processor 11 loads a program stored in the main memory 12 or the auxiliary storage device 13 into the main memory 12. The program is a program that causes the processor 11 of the image processing device 1 to realize the processing described below. The processor 11 executes the program loaded into the main memory 12 to perform various processing. The processing can also be called a function.

[0015] The main memory 12 corresponds to the main storage portion of the computer of the image processing device 1. The main memory 12 includes a nonvolatile memory area and a volatile memory area. The main memory 12 stores an operating system or a program in the nonvolatile memory area. The main memory 12 uses the volatile memory area as a work area where data is rewritten by the processor 11 as appropriate. For example, the main memory 12 includes a ROM (Read Only Memory) as a nonvolatile memory area. For example, the main memory 12 includes a RAM (Random Access Memory) as a volatile memory area.

[0016] The auxiliary storage device 13 corresponds to an auxiliary storage portion of the computer of the image processing device 1. For example, the auxiliary storage device 13 is, but is not limited to, an EEPROM (registered trademark) (Electric Erasable Programmable Read-Only Memory), an HDD (Hard Disc Drive), or an SSD (Solid State Drive). The auxiliary storage device 13 stores the above-mentioned programs, data used by the processor 11 in performing various processes, and data generated by the processes of the processor 11.

[0017] Auxiliary storage device 13 includes a character region extraction dictionary data storage area 131. Character region extraction dictionary data storage area 131 is an example of a storage unit. Character region extraction dictionary data storage area 131 stores character region extraction dictionary data. The character region extraction dictionary data is data for extracting a character region containing characters from an image. The character region extraction dictionary data may be data such as a calculation formula for extracting a character region. A character region is an area in units of one character, and is an area surrounding one character. The character region extraction dictionary data may be updated as appropriate.

[0018] The auxiliary storage device 13 includes a character recognition dictionary data storage area 132. The character recognition dictionary data storage area 132 is an example of a storage unit. The character recognition dictionary data storage area 132 stores character recognition dictionary data. The character recognition dictionary data is data related to a plurality of candidate characters for recognizing characters included in a character area. Data related to each candidate character included in the plurality of candidate characters is data for recognizing each candidate character. The data related to each candidate character may be image data, a calculation formula, or the like. The character recognition dictionary data may be updated as appropriate.

[0019] The input interface 14 is an interface for connecting the image processing device 1 to an external device. For example, the input interface 14 includes an interface for connecting the image processing device 1 to the photographing device 2. The input interface 14 is not limited to a specific standard as long as it allows the image processing device 1 to acquire data from the photographing device 2. The input interface 14 may include an interface that allows for wired connection or an interface that allows for wireless connection. The input interface 14 acquires multiple images captured by the photographing device 2. The input interface 14 is an example of an interface.

[0020] The hardware configuration of the image processing device 1 is not limited to the above configuration. The image processing device 1 allows the above components to be omitted or changed, and new components to be added, as appropriate.

[0021] The object photographed by the photographing device 2 will now be described. FIG. 2 is a diagram illustrating an example of the object 3. Object 3 includes a string of characters arranged in one direction. Here, the string is assumed to be "ABCDE," with five characters arranged in one direction. The number of characters that make up the string is not limited to this. The leftmost character in the string is assumed to be the first character, and the characters that make up the string are ordered in ascending order starting from the leftmost character. The leftmost character "A" in the string is assumed to be the first character. The second character from the left in the string, "B," is assumed to be the second character. The third character from the left in the string, "C," is assumed to be the third character. The fourth character from the left in the string, "D," is assumed to be the fourth character. The fifth character from the left in the string, "E," is assumed to be the fifth character.

[0022] (operation) Next, an example of the operation of the image processing device 1 configured as above will be described.

[0023] FIG. 3 is a flowchart illustrating an example of the procedure of image processing by the image processing device 1. The processing procedures described below are merely examples, and each process may be modified as much as possible. Furthermore, steps may be omitted, replaced, or added as appropriate depending on the embodiment. The processing performed by the processor 11 described below can also be said to be performed by a control unit including the processor 10 and the main memory 12. Therefore, the term "processor 11" may be read as "control unit."

[0024] The processor 11 acquires multiple images of a character string captured by the photographing device 2 via the input interface 14 (ACT1). In ACT1, for example, the processor 11 acquires multiple images from the photographing device 2 via the input interface 14. The processor 11 may acquire multiple images from the photographing device 2 all at once, or may acquire images one by one from the photographing device 2 in sequence. The number of multiple images may be two, or may be three or more. The processor 11 stores the multiple images in the auxiliary storage device 13. The auxiliary storage device 13 stores the multiple images.

[0025] Processor 11 repeats the processes of ACT2 to ACT7, which will be described later, for the number of images captured by imaging device 2. Processor 11 executes processes of ACT2 to ACT7 for each of the multiple images in turn. Here, an arbitrary nth image among the multiple images will be described as an example. The nth image is an example of a first image. The image immediately before the nth image among the multiple images is the n-1th image. The n-1th image is an example of a second image. If the nth image is the image to be processed, the n-1th image is a processed image.

[0026] The processor 11 extracts character regions for each character in the nth image (ACT2). In ACT2, for example, the processor 11 extracts character regions for each character in the nth image based on character recognition dictionary data. A known technique may be used for the character region extraction process. If halation, shadows, or the like occurs in a character string included in an image, the processor 11 may not be able to extract character regions for each of the characters that make up the character string. An example will be described in which the character string consists of five characters. The processor 11 may be able to extract five character regions. If halation, shadows, or the like occurs in a character string included in an image, the processor 11 may not be able to extract a character region for at least one of the five characters.

[0027] The processor 11 processes character recognition for each of all character regions extracted from the nth image (ACT3). All character regions consist of one or more character regions. In ACT3, for example, the processor 11 processes character recognition for each of all character regions based on the character recognition dictionary data. The character recognition process includes acquiring likelihood data for each character region. The likelihood data is data indicating the likelihood of each character. The likelihood data is also referred to as likelihood information. The likelihood is a numerical value, such as a probability, that indicates the likelihood that the character is included in the character region. Each character corresponds to a candidate character. Therefore, the likelihood data is data indicating the likelihood of the number of candidate characters. If the number of candidate characters set in the character recognition dictionary data is m, the likelihood data is data indicating the likelihood of each of the m characters. The larger the likelihood value, the higher the accuracy. The likelihood range is assumed to be 0 to 1, but is not limited thereto. The likelihood range may be 0 to 100. When the likelihood ranges from 0 to 1, the sum of the likelihoods of the m characters may or may not be 1.

[0028] Processor 11 acquires likelihood data for each character region in the nth image based on the character recognition of all character regions. For example, if four character regions are extracted from the nth image, processor 11 acquires likelihood data for each of the four character regions. The character recognition process may use known techniques.

[0029] The processor 11 extracts one or more local features from the nth image (ACT4). In ACT4, in one example, the processor 11 extracts local features from a character string included in the nth image. In another example, the processor 11 extracts local features from a portion different from the character string included in the nth image. In this example, the portion different from the character string may be a background portion of an object or the like included in the nth image. The local features may be local features extracted from a pattern of a portion different from the character string included in the object. A known technique may be used for the extraction process of local features.

[0030] Processor 11 determines whether the nth image is the second or subsequent image (ACT5). If the nth image is the second or subsequent image (ACT5, YES), the process transitions from ACT5 to ACT6. If the nth image is not the second or subsequent image (ACT5, NO), the process transitions from ACT5 to ACT7. The fact that the nth image is not the second or subsequent image corresponds to the nth image being the first image.

[0031] The processor 11 processes image matching (ACT6). In ACT6, for example, the processor 11 processes image matching between different images. Image matching includes a process of adjusting the positional relationship of one of the different images to that of the other image. Adjusting the positional relationship of one image to that of the other image includes adjusting the position of a character string included in one image to the position of a character string included in the other image. Image matching includes feature point matching based on local features. In this example, the processor 11 processes feature point matching of one or more local features extracted from the different images. The processor 11 acquires a homography matrix based on the image matching. The homography matrix may be a calculation formula or the like for adjusting the positional relationship of one of the different images to that of the other image. The processor 11 may acquire the homography matrix using information indicating the position of the camera 2 that captures the character string. The position of the camera 2 may be the distance of the camera 2 from the character string. The processor 11 may acquire the homography matrix using information indicating the orientation of the camera 2 that captures the character string. The orientation of the camera 2 may be the orientation of the camera 2 with respect to the character string. For example, the processor 11 can process image matching between the nth image and the n-1th image. The processor 11 can process feature point matching between one or more local features extracted from the nth image and one or more local features extracted from the n-1th image. The processor 11 can acquire a projective transformation matrix for adjusting the positional relationship of the n-1th image to the positional relationship of the nth image based on the image matching. Note that the processor 11 may acquire a projective transformation matrix to be applied to the nth image instead of acquiring a projective transformation matrix to be applied to the n-1th image.

[0032] The processor 11 determines identical character regions based on image matching between different images (ACT7). An identical character region is a character region that is included in two or more of the multiple images. A character region that is included in two or more images is a character region related to the same character among one or more characters that make up a character string. Hereinafter, an identical character region is also referred to as a first character region. Determining the first character region includes determining whether or not the first character region exists.

[0033] In ACT7, for example, processor 11 applies a projective transformation matrix obtained based on image matching between different images to one image. Processor 11 performs projective transformation to match one image to the positional relationship of the other image, thereby adjusting the positional relationship of one image to the positional relationship of the other image. Processor 11 also transforms the positions of one or more character regions extracted from one image in accordance with this projective transformation. Processor 11 may match the positional relationship of one image to the positional relationship of the other image, or may match the positional relationship of the other image to the positional relationship of the one image. The reference image is arbitrary. Processor 11 compares the positional relationship between one or more character regions extracted from one image and one or more character regions extracted from the other image based on the projective transformation of one image. The positional relationship includes overlapping relationships. For example, processor 11 can perform projective transformation to match the positional relationship of the (n-1)th image to the positional relationship of the nth image. Processor 11 can adjust the positional relationship of the (n-1)th image to the positional relationship of the nth image. The processor 11 can compare the positional relationship between one or more character regions extracted from the (n-1)th image and one or more character regions extracted from the nth image based on the projective transformation.

[0034] The processor 11 identifies one or more character regions constituting a character string based on the determination of the first character region. For example, the processor 11 determines whether one or more character regions extracted from one image and one or more character regions extracted from the other image are identical based on their relative positions. The determination of identical character regions includes determining which character regions between one image and the other image are identical. Determining which character regions are identical includes determining a first character region included in one or more images among the multiple images based on image matching of the multiple images. The processor 11 may determine the first character region from one image including the first character region, or may determine the first character region from two or more images including the first character region. For example, the processor 11 determines, based on the comparison, character regions included in two images that are positioned to overlap as the first character region. In this example, the processor 11 may determine the first character region based on the degree of overlap between the character regions included in the two images. The degree of overlap is the degree of overlap in the positions of the two images. The overlap threshold for determining a first character region may be set as appropriate. Processor 11 may not determine a first character region, or may determine one or more first character regions. Processor 11 identifies one or more character regions constituting a character string based on a comparison of the positional relationship between one or more character regions extracted from one image and one or more character regions extracted from the other image. Identifying one or more character regions constituting a character string includes identifying an arrangement of one or more character regions constituting the character string and identifying the number of characters constituting the character string based on the identification of one or more character regions constituting the character string. For example, processor 11 may identify one or more character regions constituting a character string based on the same character region determination between the (n-1)th image and the nth image.

[0035] The processor 11 repeats the above-described processes of ACT2 to ACT7 for the number of images captured by the image capturing device 2, thereby realizing the following processes. By repeating the process of ACT2, the processor 11 can extract character regions for each character from each of the multiple images. Processor 11 performs character recognition for all character regions extracted from each of the multiple images by repeating the process of ACT3. Processor 11 can obtain likelihood data for each character region for each of the multiple images based on the character recognition. By repeating the process of ACT4, the processor 11 can extract one or more local features from each of the multiple images. The processor 11 can process image matching for a plurality of images by repeating the process of ACT6. By repeating the process of ACT7, processor 11 can determine a first character region based on image matching of multiple images. Processor 11 can identify one or more character regions that make up a character string based on the determination of the first character region. The one or more character regions that make up a character string may all be first character regions. The one or more character regions that make up a character string may not include any first character regions. In this case, the one or more character regions that make up a character string are all second character regions, different from the first character regions. The second character region is a character region that is included in one image of the multiple images. The second character region is a character region that is extracted from only one image of the multiple images. The one or more character regions that make up a character string may include both a first character region and a second character region.

[0036] The processor 11 identifies characters for each character region that constitutes the character string based on the identification of one or more character regions that constitute the character string (ACT8). In ACT8, for example, the processor 11 identifies characters for each character region that constitutes the character string based on likelihood data.

[0037] A case where one or more character regions constituting a character string include a first character region will be described. When one or more character regions constituting a character string include one or more first character regions, processor 11 performs the following character identification process for each of the one or more first character regions. For each of the one or more first character regions, processor 11 uses two or more pieces of likelihood data for the first character region. The two or more pieces of likelihood data for the first character region are multiple pieces of likelihood data related to the first character region acquired from two or more images including the first character region. Processor 11 identifies a character in the first character region based on the two or more pieces of likelihood data for the first character region acquired from the two or more images. For example, processor 11 sums the likelihood of each character indicated by each of the two or more pieces of likelihood data for each character and calculates a value based on the sum for each character. For example, the value based on the sum is a sum or an average value, but is not limited to these. Since the value based on the sum is a value obtained by summing two or more likelihoods, it is a numerical value representing the likelihood, similar to the likelihood. When two or more pieces of likelihood data are two pieces of likelihood data, the value based on the sum is a value obtained by summing the two likelihoods. Processor 11 identifies the character associated with the most likely value among the values ​​based on the sum for each character as the character of the first character region. For example, the most likely value is the maximum value.

[0038] A case where one or more character regions constituting a character string include a second character region will be described. When one or more character regions constituting a character string include one or more second character regions, processor 11 performs the following character identification process for each of the one or more second character regions. Processor 11 uses likelihood data of the second character region for each of the one or more second character regions. The likelihood data of the second character region is likelihood data related to the second character region acquired from an image including the second character region. Processor 11 identifies a character in the second character region based on the likelihood data of the second character region acquired from the image. Processor 11 identifies the character associated with the most likely value among the likelihoods of each character indicated by the likelihood data as a character in the second character region. For example, the most likely value is the maximum value.

[0039] Processor 11 identifies the character associated with the highest likelihood value for each character region as the character for each character region. Processor 11 may generate data consisting of a numerical value representing the likelihood of each character for each character region constituting the character string. For the first character region, the numerical value representing the likelihood of each character is a value based on the sum of the likelihoods for each character obtained based on two or more likelihood data. For the second character region, the numerical value representing the likelihood of each character is the likelihood of each character obtained based on one likelihood data.

[0040] Processor 11 identifies a character string based on the identification of characters for each character region that constitutes the character string. For example, processor 11 acquires one or more characters that constitute the character string based on the identification of characters for each character region that constitutes the character string. Processor 11 identifies a character string in which the characters for each character region are arranged.

[0041] As described above, processor 11 can identify characters for each character area that constitutes a character string based on the identification of one or more character areas that constitute the character string. Processor 11 can identify a character string based on the identification of characters for each character area that constitutes the character string. These processes are examples of identifying a character string based on the identification of one or more character areas that constitute the character string.

[0042] An example of an image captured by the image capturing device 2 will be described. FIG. 4 is a diagram illustrating an example of an image captured by the image capturing device 2. As shown in FIG. FIG. 4 shows the (n-1)th image and the nth image captured by the image capture device 2 of the character string included in the object 3 illustrated in FIG. In the (n-1)th image, halation occurs at the third and fourth characters of the string of five characters. In the nth image, halation occurs at the first character of the string of five characters.

[0043] An example of the process of extracting the character region in the above-mentioned ACT2 by the image processing device 1 will be described. FIG. 5 is a diagram illustrating an example of the extraction process of a character region by the image processing device 1. In FIG. Processor 11 extracts three character regions, the first, second, and fifth characters, from a string of five characters in the (n-1)th image. Processor 11 does not extract two character regions, the third and fourth characters, due to halation that occurs in the (n-1)th image. Processor 11 extracts four character regions, the second, third, fourth, and fifth characters, from a string of five characters in the nth image. Processor 11 does not extract the single character region of the first character due to halation occurring in the nth image. An example of the character recognition process in the above-mentioned ACT3 by the image processing device 1 will be described with reference to FIG. The processor 11 performs character recognition processing on each of the three character regions extracted from the (n-1)th image, and obtains likelihood data for each of the three character regions. The processor 11 performs character recognition processing on each of the four character regions extracted from the n-th image, and obtains likelihood data for each of the four character regions.

[0044] An example of the process of determining identical character regions in ACT7 by the image processing device 1 will be described. FIG. 6 is a diagram illustrating the process of determining identical character regions by the image processing device 1. In FIG. Processor 11 performs projective transformation on the (n-1)th image to match the positional relationship of the (n-1)th image with the positional relationship of the (n-1)th image, thereby matching the positional relationship of the (n-1)th image with the positional relationship of the (n-1)th image. At the same time, processor 11 also transforms the positions of the three character regions extracted from the (n-1)th image in accordance with this projective transformation. Processor 11 may match the positional relationship of the (n-1)th image with the positional relationship of the (n-1)th image, or may match the positional relationship of the (n-2)th image with the positional relationship of the (n-1)th image. The reference image is arbitrary. Based on the projective transformation of the (n-1)th image, processor 11 compares the positional relationship between the three character regions extracted from the (n-1)th image and the four character regions extracted from the (n-2)th image, and determines which character regions in the (n-1)th image and the (n-1)th image are identical. Processor 11 identifies the five character regions that make up the character string based on a comparison of the positional relationships between the (n-1)th image and the (n-1)th image that have undergone projective transformation. Processor 11 identifies the number of characters that make up the character string based on the identification of the five character regions that make up the character string.

[0045] An example of the process of specifying characters for each character area in ACT8 by the image processing device 1 will be described. FIG. 7 is a diagram illustrating the process of specifying characters for each character area performed by the image processing device 1. In FIG. Here, the description will be made assuming that the number of images is two. The Nth image is the second image. The N-1th image is the first image. The processor 11 identifies the characters for each character area that constitutes the character string based on the identification of the five character areas that constitute the character string.

[0046] The following describes how to identify a character in a character region for the first character that constitutes a character string. The character region for the first character is a second character region included in the first image. Processor 11 identifies a character in the first character region based on likelihood data for the first character region included in the first image. Processor 11 identifies the character "A," which is associated with the highest likelihood value among the likelihoods of each character indicated by the likelihood data, as the character in the first character region.

[0047] The following describes how to identify a character in a character region for a second character that constitutes a character string. The character region for the second character is a first character region included in the first image and the second image. Processor 11 identifies a character in the character region for the second character based on two pieces of likelihood data for the character region for the second character obtained from the first image and the second image. Processor 11 sums the likelihoods of each character indicated by the two pieces of likelihood data for each character, and determines a value based on the sum for each character. Processor 11 identifies the character "B" associated with the most likely value among the values ​​based on the sum for each character as a character in the character region for the second character.

[0048] The following describes how to identify a character in a third character region that constitutes a character string. The third character region is a second character region included in the second image. Processor 11 identifies a character in the third character region based on likelihood data for the third character region included in the second image. Processor 11 identifies the character "C," which is associated with the highest likelihood value among the likelihoods of each character indicated by the likelihood data, as the character in the third character region.

[0049] The following describes how to identify a character in a fourth character region that constitutes a character string. The fourth character region is a second character region included in the second image. Processor 11 identifies a character in the fourth character region based on likelihood data for the fourth character region included in the second image. Processor 11 identifies the character "D," which is associated with the highest likelihood value among the likelihoods of each character indicated by the likelihood data, as the character in the fourth character region.

[0050] The following describes how to identify a character in a fifth character region that constitutes a character string. The fifth character region is a first character region included in the first and second images. Processor 11 identifies a character in the fifth character region based on two pieces of likelihood data for the fifth character region acquired from the first and second images. Processor 11 sums the likelihoods of each character indicated by each of the two pieces of likelihood data for each character, and determines a value based on the sum for each character. Processor 11 identifies the character "E" associated with the most likely value among the values ​​based on the sums for each character as the character in the fifth character region.

[0051] As described above, processor 11 identifies the character associated with the highest likelihood value for each character region as the character for that character region. Processor 11 identifies the character string "ABCDE" based on the identification of the characters for each character region that make up the character string.

[0052] (effect) An image processing device according to an embodiment includes an interface and a processor. The interface acquires a plurality of images captured by an image capture device. The processor extracts a character region for each character from each of the plurality of images. The processor determines a first character region included in one or more of the plurality of images based on image matching of the plurality of images. The processor identifies one or more character regions constituting a character string based on the determination of the first character region. The processor identifies the character string based on the identification of the one or more character regions constituting the character string. Therefore, even if the image processing device cannot identify a character string from a single image due to the influence of halation, shadows, etc., it can identify the character string from multiple images. This allows the image processing device to improve the accuracy of character string recognition. For example, in a warehouse, improving the accuracy of character string recognition from cardboard boxes wrapped in transparent sheets when receiving them improves receiving efficiency. For example, in a restaurant, improving the accuracy of character string recognition from menus in clear folders improves the accuracy of menu translation using electronic devices.

[0053] The processor extracts one or more local features from each of the plurality of images, and the image matching includes feature point matching based on the local features. This allows the image processing device to improve the accuracy of image matching, thereby improving the accuracy of character string recognition based on multiple images.

[0054] The processor acquires likelihood data indicating the likelihood of each character for each character region for each of the multiple images. The processor identifies characters for each character region that make up the character string based on the likelihood data. The processor identifies the character string based on the identification of characters for each character region that make up the character string. Therefore, the image processing device can improve the accuracy of identifying characters for each character region using the likelihood data, thereby improving the accuracy of recognizing character strings.

[0055] If one or more character regions constituting the character string include a first character region, the processor identifies a character in the first character region based on two or more pieces of likelihood data for the first character region obtained from two or more images. Therefore, for character regions included in two or more images, the image processing device identifies characters based on two or more likelihood data instead of one likelihood data, thereby improving the accuracy of identifying characters, and thereby improving the accuracy of recognizing character strings.

[0056] If one or more character regions constituting the string include a second character region contained in one of the multiple images, the processor identifies a character in the second character region based on likelihood data of the second character region obtained from one of the images. Therefore, the image processing device can identify characters for a character region included in one image based on one likelihood data. Since the image processing device does not necessarily need multiple images containing the character region to identify the characters in the character region, the processing load for recognizing the character string can be reduced.

[0057] (Variation) While the example in which processor 11 processes multiple images to identify a character string has been described, the number of images may or may not be predetermined. In the latter case, processor 11 may execute the processes in ACT2 to ACT7 for a first predetermined number of images before executing the process in ACT8. The first predetermined number is two or more, but can be set as appropriate. In this example, processor 11 may add a second predetermined number of images for executing the processes in ACT2 to ACT7, depending on the identification accuracy in the process in ACT8. The second predetermined number is one or more, but can be set as appropriate. Processor 11 may determine whether to terminate or continue the character identification process for each character region, depending on the highest likelihood value for each character region in the process in ACT8. Terminating the character identification process for each character region includes not adding the second predetermined number of images. Continuing the character identification process for each character region includes adding the second predetermined number of images. When all character regions constituting the character string satisfy the condition, processor 11 terminates the character identification process for each character region. For example, the condition includes that the most likely value is equal to or greater than a predetermined value. The predetermined value can be set as appropriate. If at least one character area among all character areas constituting the character string does not satisfy the condition, the processor 11 continues the character identification process for each character area. When continuing the character identification process for each character area, the processor 11 acquires a second predetermined number of images. The processor 11 performs processes ACT2 to ACT7 on the acquired second predetermined number of images. The processor 11 performs process ACT8 based on multiple images including the first predetermined number of images and the second predetermined number of images. The processor 11 repeatedly adds the second predetermined number of images until all character areas constituting the character string satisfy the condition. In this way, the multiple images may be the first predetermined number, or the first predetermined number plus a second predetermined number added one or more times. The image processing device 1 stops processing images necessary to identify the character string, thereby reducing the processing load. The processor 11 may also process images captured by the imaging device 2 in real time.In this example, when the processor 11 ends the character specification process for each character area, the processor 11 may output an instruction to the photographing device 2 to end photographing.

[0058] The above-mentioned extracting a character region includes the detection and acquisition of a character region, etc. The above-mentioned acquiring likelihood data includes the calculation of likelihood, etc. The above-mentioned extracting local features includes the detection and acquisition of local features, etc. The above-mentioned determining identical character regions includes the detection, acquisition, determination, estimation, etc. of identical character regions, etc. The above-mentioned specifying one or more character regions constituting a character string includes the detection, acquisition, determination, estimation, etc. of a character region, etc. The above-mentioned specifying a character for each character region constituting a character string includes the detection, acquisition, determination, estimation, etc. of a character, etc. The above-mentioned specifying a character string includes the detection, acquisition, determination, estimation, etc. of a character string.

[0059] The image processing device may be realized by a single device such as the image processing device 1, or may be realized by a plurality of devices with distributed functions. The image processing device is not limited to a device that does not have a photographing function, such as the image processing device 1. The image processing device may be realized by a device that has a photographing function.

[0060] The program may be transferred in a state where it is stored in a device, or in a state where it is not stored in a device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The form of the recording medium is not important as long as it is a medium that can store the program and is computer-readable, such as a CD-ROM or a memory card.

[0061] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0062] 1...image processing device, 2...photographing device, 3...object, 11...processor, 12...main memory, 13...auxiliary storage device, 14...input interface, 131...dictionary data storage area for character region extraction, 132...dictionary data storage area for character recognition.

Claims

1. an interface for acquiring a plurality of images captured by an imaging device; a processor that extracts character regions, which are regions for individual characters, from characters that make up a character string included in each of the multiple images, determines the presence or absence of a first character region, which is a character region for the same character included in two or more of the multiple images, based on image matching of the multiple images, identifies an arrangement of one or more character regions that make up the character string based on the determination of the presence or absence of the first character region, and identifies characters for each character region that make up the character string based on the identification of the arrangement of one or more character regions that make up the character string; An image processing device comprising:

2. The processor extracts one or more local features from each of the plurality of images; the image matching includes feature point matching based on the local feature amount; The image processing device according to claim 1 .

3. 3. The image processing device according to claim 1, wherein the processor acquires likelihood data indicating the likelihood of each character for each character area for each of the plurality of images, identifies characters for each character area that constitute the character string based on the likelihood data, and identifies the character string based on the identification of characters for each character area that constitute the character string.

4. 4. The image processing device according to claim 3, wherein, when one or more character regions constituting the character string include the first character region, the processor identifies the character in the first character region based on two or more likelihood data of the first character region obtained from two or more images.

5. 5. The image processing device according to claim 3, wherein when one or more character regions constituting the character string include a second character region that is different from the first character region and is a character region extracted in only one image among the plurality of images, the processor identifies a character in the second character region based on likelihood data of the second character region obtained from the one image.

6. On the computer, a function of acquiring a plurality of images taken by the imaging device; a function of extracting character regions, which are regions of individual characters, from characters constituting a character string included in each of the plurality of images; a function of determining the presence or absence of a first character area, which is a character area relating to the same character included in two or more images among the plurality of images, based on image matching of the plurality of images; a function of identifying a sequence of one or more character regions that constitute the character string based on the determination of the presence or absence of the first character region; a function of identifying characters for each character region that constitutes the character string based on identification of an arrangement of one or more character regions that constitute the character string; An image processing program for executing the above.

Images