Image processing apparatus, image processing method, and computer program

Abstract

To enable a detection of a corner of a document even if the entire document is not imaged in an image.SOLUTION: In an image processing device 11, a region identifying unit identifies, from an image containing a background and a document, a first region representing the document, and a second region representing an object other than the document and the background. A feature point identifying unit identifies, on the basis of the first region and the second region, multiple feature points containing at least any one of a point that is a corner of the document, a point contacting an external edge of the second region at an external edge of the first region, and a point contacting an external edge of the image at the external edge of the first region. A determining unit determines whether or not the adjoining two feature points among the multiple feature points are on the same side in the document. A corner identifying unit identifies four corners of the document on the basis of the adjoining two feature points determined as being on the same side.SELECTED DRAWING: Figure 7

Description

Technical Field

[0005]

[0001] The present disclosure relates to an image processing apparatus, an image processing method, and a computer program.

Background Art

[0002] Patent Document 1 describes an image processing apparatus capable of detecting the outer contour of a document. Specifically, in the image processing apparatus, an image in which the document is reflected is acquired from an imaging device or the like. Thereafter, four virtual straight lines are translated parallel to the center starting from each vertex of the image. Each virtual straight line is inclined at a predetermined angle with respect to two sides orthogonal to each other in the image. Further, in the image processing apparatus, intersections between each virtual straight line and the outer contour are acquired, and corners of the outer contour of the document are acquired based on each intersection.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the image processing apparatus of Patent Document 1 has a problem that it is difficult to detect the corners of a document if the entire document is not reflected in the image. Specifically, in a conversation among a plurality of people, the document may be referred to. When detecting the corners of the document by the image processing apparatus, each person needs to pay attention not to place their hand on the document. In addition, it is necessary to adjust the position of the image processing apparatus so that the entire area of the document falls within the angle of view of the imaging device, or to pay attention so that the corners of the document are not folded.

[0005] The present disclosure has been made in view of the above situation, and an object thereof is to provide a technology capable of detecting the corners of a document even when the entire document is not reflected in the image.

Means for Solving the Problems

[0006] An image processing apparatus according to one aspect of the present disclosure comprises a region identification unit, a feature point identification unit, a determination unit, and a corner identification unit. The region identification unit identifies a first region representing the document and a second region representing objects other than the document and the background from an image including the background. The feature point identification unit identifies a plurality of feature points based on the first region and the second region, including at least one of the following: a point that is a corner of the document, a point on the outer edge of the first region that touches the outer edge of the second region, and a point on the outer edge of the first region that touches the outer edge of the image. The determination unit determines whether two adjacent feature points among the plurality of feature points lie on the same side of the document. The corner identification unit identifies the four corners of the document based on the two adjacent feature points determined to be on the same side.

[0007] Image processing methods relating to other aspects of the present disclosure include the steps of: identifying a first region representing a document and a second region representing objects other than the document and the background from an image including a background and a document; identifying a plurality of feature points based on the first region and the second region, which are any of the following: a point that is a corner of the document, a point on the outer edge of the first region that touches the outer edge of the second region, and a point on the outer edge of the first region that touches the outer edge of the image; determining whether two adjacent feature points among the plurality of feature points lie on the edge of the document; and identifying the four corners of the document based on the two adjacent feature points determined to be on the edge.

[0008] A computer program relating to yet another aspect of the present disclosure causes the computer to perform the steps of: identifying from an image including a background and a document a first region representing the document and a second region representing objects other than the document and the background; identifying a plurality of feature points based on the first region and the second region that are any of the following: points that are corners of the document, points on the outer edge of the first region that touch the outer edge of the second region, and points on the outer edge of the first region that touch the outer edge of the image; determining whether two adjacent feature points among the plurality of feature points lie on the edge of the document; and identifying the four corners of the document based on the two adjacent feature points determined to be on the edge. [Effects of the Invention]

[0009] According to this disclosure, it is possible to provide a technology that can detect the corners of a document even when the entire document is not visible in the image. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an information processing device according to one embodiment of the present disclosure. [Figure 2] This figure shows the detailed configuration of the information processing device shown in Figure 1. [Figure 3] This figure shows the block configuration of the processing unit shown in Figure 2. [Figure 4] This figure shows the detailed configuration of the server device shown in Figure 1. [Figure 5] Figure 3 is the main flowchart showing the processing steps of the processing unit. [Figure 6] This figure shows the images represented by the unprocessed image data captured in the first and second situations. [Figure 7] Figure 5 is a flowchart showing the detailed processing steps for step S102. [Modes for carrying out the invention]

[0011] Embodiments of this disclosure will be described with reference to the drawings. In the drawings, the same or corresponding parts will be denoted by the same reference numerals and will not be repeated in the description.

[0012] Figure 1 shows an information processing device 100 according to one embodiment of the present disclosure. As shown in Figure 1, the information processing device 100 can communicate data with a server device 300 via a communication network 200. The communication network 200 is the Internet, a wired LAN (Local Area Network), or a wireless LAN. The communication network 200 may be a combination of two or more selected from the Internet, a wired LAN, and a wireless LAN.

[0013] Figure 2 shows a detailed configuration of the information processing device 100 shown in Figure 1. As shown in Figure 2, the information processing device 100 further comprises an imaging unit 12, a display unit 13, a connector unit 14, and a communication interface unit 15. Hereafter, the "communication interface unit" will be referred to as the "communication IF unit".

[0014] The image processing device 11 is, for example, the main unit of a personal computer. The image processing device 11 comprises a storage unit 111 and a processing unit 112.

[0015] The storage unit 111 is, for example, a non-temporary computer-readable storage medium. More specifically, the storage unit 111 includes RAM (Random Access Memory), ROM (Read Only Memory), and / or flash memory. The storage unit 111 stores an image processing program 113. The image processing program 113 defines the processing procedure of the processing unit 112. The image processing program 113 is also an example of a "computer program" in this disclosure. The image processing program 113 may be a single program or a collection of multiple program modules.

[0016] The processing unit 112 is, for example, a CPU (Central Processing Unit) or a microcomputer. By executing the image processing program 113, the processing unit 112 functions as an image acquisition unit 112A, a specifying unit 112B, a deformation unit 112C, a character recognition unit 112D, and a composition unit 112E. As a result, the processing unit 112 controls the imaging unit 12, the display unit 13, the storage medium 16 that can be connected to the connector unit 14, and the communication IF unit 15. The processing procedure of the image processing program 113 and the control of each unit will be described in detail later.

[0017] The imaging unit 12 is an imaging device such as a camera. The imaging unit 12 captures an image within its own angle of view and generates image data indicating the captured image. Note that the imaging unit 12 can transmit the image data to the image processing apparatus 11 via a communication cable or a wireless link.

[0018] The display unit 13 is a display device such as a liquid crystal display. The display unit 13 displays various images on the screen. The display unit 13 can communicate with the image processing apparatus 11 via a communication cable or the like.

[0019] The connector unit 14 and the communication IF unit 15 are provided in the image processing apparatus 11.

[0020] The connector unit 14 is a receptacle to which the storage medium 16 can be connected. Specifically, the storage medium 16 is, for example, a USB memory. The storage medium 16 can store various data.

[0021] The communication IF unit 15 is an interface that enables data communication with the server device 300 via the communication network 200 (see Figure 1). Specifically, the communication IF unit 15 sends request data to the communication network 200 with the server device 300 as the destination. In this embodiment, the request data is information for requesting object recognition processing by AI (Artificial Intelligence) (hereinafter simply referred to as "object recognition processing"). The request data includes image data that is the target of the object recognition processing. The communication IF unit 15 further receives response data from the communication network 200 with the information processing device 100 as the destination.

[0022] Figure 4 is a diagram showing the detailed configuration of the server device 300 shown in Figure 1. As shown in Figure 4, the server device 300 stores a communication interface unit 31, a storage unit 32, and a processing unit 33.

[0023] The communication interface unit 31 is an interface that enables data communication with the information processing device 100 via the communication network 200. The communication interface unit 31 receives request data from the communication network 200 with the server device 300 as the destination. The communication interface unit 31 sends response data to the communication network 200 with the information processing device 100 as the destination.

[0024] The storage unit 32 includes RAM (Random Access Memory), ROM (Read Only Memory), and / or flash memory. The storage unit 32 stores the object model database 321. Hereinafter, "object model database" will be referred to as "object model DB".

[0025] The processing unit 33 is, for example, a CPU or a microcomputer. The processing unit 33 performs machine learning for object recognition processing. That is, the processing unit 33 acquires image data. The image data represents an image containing various objects. The processing unit 33 extracts local features of each object contained in the image shown by the image data. The processing unit 33 registers the local features and the identification information that identifies the object in the object model DB 321, linking them together.

[0026] The processing unit 33 may receive request data from the information processing device 100 through the communication IF unit 31. The processing unit 33 starts object recognition processing triggered by the receipt of request data. Specifically, the processing unit 33 extracts local features from the image data contained in the received request data. Based on the extracted local features, the processing unit 33 searches the object model DB 321 to recognize the objects depicted in the image data. For object recognition processing, for example, panoptic segmentation can be used. In panoptic segmentation, each pixel constituting the image is individually assigned a label that can identify an object. Furthermore, for countable objects, objects can be distinguished pixel by pixel using Detectron2 or the like.

[0027] Upon completion of the object recognition process, the processing unit 33 generates response data. The response data includes information indicating the recognized object. The destination for the response data is specified as the information processing device 100. The processing unit 33 sends the created response data to the communication network 200 via the communication IF unit 31.

[0028] In this embodiment, the information processing device 100 is used to digitize documents that are referenced in conversation. The documents are rectangular in shape when they are free from folds, cuts, curls, etc. For example, in a pharmacy, a pharmacist refers to documents for patients to provide medication guidance to them. A form is an example of such a document. The main surface of the form contains descriptions of the prescribed medications, etc. When the information processing device 100 is used in a pharmacy, the image processing device 11 and the display unit 13 are installed on a counter or table where the forms are placed during medication guidance, etc. The imaging unit 12 is installed so as to be able to photograph the forms on the counter or table.

[0029] Figure 5 is the main flowchart of the processing unit 112 shown in Figure 3. As shown in Figure 5, the processing of the processing unit 112 includes steps S101 to S106.

[0030] Step S101 is an image acquisition process. In step S101, the processing unit 112 functions as an image acquisition unit 112A and acquires image data (hereinafter referred to as "pre-processing image data") from the imaging unit 12. The pre-processing image data includes an image of the background and the document (form). By processing the pre-processing image data from the imaging unit 12, the processing unit 112 can easily digitize documents during the interaction. However, if the pre-processing image data stored in the storage medium 16 is to be processed, it is necessary to transfer the pre-processing image data from the storage medium 16 to the processing unit 112, making it difficult to digitize documents during the interaction.

[0031] Step S102 is a specific process. In step S102, the processing unit 112 functions as a specific unit 112B and identifies the document area from all pixels constituting the image shown by the pre-processing image data (hereinafter referred to as the "pre-processing image"). The document area is the area of ​​pixels that constitute the document. The document area is also an example of the "first area" in this disclosure. The processing unit 112 further identifies the pixel positions that indicate the four corners of the document in the pre-processing image.

[0032] Step S103 is a deformation process. In step S103, the processing unit 112 functions as a deformation unit 112C and generates a rectangular image by deforming the document area into a rectangle based on the pixel positions indicating the four corners of the document.

[0033] Step S104 is character recognition processing. In step S104, the processing unit 112 functions as a character recognition unit 112D and performs character recognition processing on the document area. Through character recognition processing, the processing unit 112 converts each character contained in the document area into character data represented by a predetermined character code.

[0034] Step S105 is a synthesis process. In step S105, the processing unit 112 functions as a synthesis unit 112E and synthesizes the recognition results of the character recognition unit 112D into a rectangular image. The recognition results are character data converted by the character recognition unit 112D. Specifically, in the rectangular image, the recognition results are synthesized into the areas where each character was laid out before the character recognition process.

[0035] As a result of steps S101 to S105, processed image data is generated. The processed image data shows a rectangular image in which the recognition result is composited onto the document area. Step S106 is executed after step S105. Step S106 is a display process. In step S106, the processing unit 112 displays the rectangular image in which the recognition result is composited on the display unit 13 based on the processed image data. The processed image data can also be used for saving or printing. In the saving process, the processing unit 112 saves the processed image data to the storage unit 111. In the printing process, the processing unit 112 prints the rectangular image based on the processed image data to a printer (not shown).

[0036] Incidentally, during the conversation, the document may be placed in, for example, the first, second, and third situations. In the first situation, the document is outside the field of view of the imaging unit 12. In the second situation, a hand is placed on the document. In the third situation, a fold is made in the corner of the document.

[0037] Figure 6 shows the image represented by the pre-processing image data captured in the first and second situations, i.e., the pre-processing image 400. The pre-processing image 400 is a full-color image, etc., and has pixel values ​​for each coordinate value in a two-dimensional image coordinate system. As shown in Figure 6, the pre-processing image 400 includes a document 41, a background 42, and unwanted objects 43.

[0038] Document 41 is a document referred to in the dialogue. Background 42 is the surface on which Document 41 is placed. In detail, Background 42 is a counter or table, etc. Unwanted objects 43 are objects other than Document 41 and Background 42 in the pre-processing image 400. In embodiments, objects are a concept that includes the human body. In the example of Figure 6, Unwanted object 43 is a hand placed on Document 41. Note that Unwanted objects 43 may not be visible in the pre-processing image 400. Also, Unwanted objects 43 are an example of "objects other than Document and Background" in this disclosure.

[0039] In Figure 3, a dashed line 411 is shown in relation to the first situation. The dashed line 411 indicates the portion of the document 41 that is outside the field of view of the imaging unit 12 (see Figure 2). In this embodiment, the field of view is the area enclosed by the outer edge 414 of the pre-processing image 400. Figure 6 further shows unwanted objects 43 in relation to the second situation. Due to the unwanted objects 43, a portion of the document 41 is not visible in the pre-processing image 400. Note that folds related to the third situation are not shown in Figure 6. In the third situation, due to the folding caused by the folds, a portion of the document 41 is not visible in the pre-processing image 400.

[0040] The image processing device 11 can identify the four corners of the document 41 by the following specific processing, even in situations where the entire main surface of the document 41 is not visible in the pre-processing image 400, such as the first, second, and third situations.

[0041] Figure 7 is a flowchart showing the detailed processing steps of the specific process (step S102) shown in Figure 5. As shown in Figure 7, the specific process performed by the processing unit 112 includes steps S201 to S205. The specific process will be explained in detail below with reference to Figures 6 and 7.

[0042] Step S201 is a region identification process. The processing unit 112 that executes step S201 is an example of a "region identification unit" in this disclosure. In step S201, the processing unit 112 identifies a document region 412 and an unwanted object region 431 from the pre-processing image 400. The document region 412 is a region where pixels representing the document 41 are distributed. The unwanted object region 431 is a region where pixels representing unwanted objects 43 are distributed. The document region 412 and the unwanted object region 431 are examples of a "first region" and a "second region" in this disclosure.

[0043] In detail, in step S201, the processing unit 112 first performs a skin-color region identification process to identify the unwanted region 431. In the skin-color region identification process, the processing unit 112 identifies pixels included in a first identification range from among all pixels constituting the pre-processed image 400 as the unwanted region 431. The first identification range is a range of pixel values ​​predetermined as skin color. Because the unwanted region 431 can be identified by the skin-color region identification process, no additional devices are required for the information processing device 100.

[0044] In step S201, the processing unit 112 further determines the group of pixels to be processed, which is obtained by removing the pixels included in the unwanted area 431 from all the pixels that make up the pre-processing image. From the group of pixels to be processed, the processing unit 112 identifies the pixels included in the second specified range. The second specified range is a range of pixel values ​​predetermined as the color of the document area 412. The color of the document area 412 is, for example, white. The processing unit 112 identifies the group of pixels included in the second specified range as the document area 412. Note that the document 41 contains text and / or photographs. However, to identify the four corners of the document 41, it is sufficient to identify the outer edge 413 of the document area 412. Therefore, the processing unit 112 may ignore the text or photographs recorded inside the outer edge 413.

[0045] Step S202 is a reference point determination process. The processing unit 112 that executes step S202 is an example of a "reference point determination unit" in this disclosure. In step S202, the processing unit 112 obtains a plurality of feature points P1 at the outer edge 413 of the document area 412. Specifically, the processing unit 112 generates a binary image for all pixels constituting the document area 412. The processing unit 112 further performs a convex hull operation on the binary image and then applies the Douglas-Peucker algorithm. As a result, the processing unit 112 obtains a plurality of feature points P1 at the outer edge 413. Figure 6 shows an example of six feature points P1.

[0046] The processing unit 112 determines a reference point P2 based on the acquired feature points P1. Therefore, the amount of computation can be reduced compared to when the reference point is determined based on all pixels of the document area 412. The reference point P2 is the reference point for the circumferential direction θ required in the determination process (step S204) described later. In detail, the reference point P2 is contained within a polygon whose vertices are the multiple feature points P1. An example of a reference point P2 is the centroid of the multiple feature points P1. Other examples of reference point P2 are as follows. Each of the multiple feature points P1 has a coordinate value in the image coordinate system. The coordinate value is a combination of x-coordinate value and y-coordinate value. In this case, the x-coordinate value of the reference point P2 is the average value of each x-coordinate value of the multiple feature points P1. The y-coordinate value of the reference point P2 is the average value of each y-coordinate value of the multiple feature points P1.

[0047] Step S203 is a feature point identification process. The processing unit 112 that performs step S203 is an example of a "feature point identification unit" in this disclosure. In step S203, the processing unit 112 identifies whether a plurality of feature points P1 is a first feature point P11, a second feature point P12, or a third feature point P13, based on the document area 412 and the unwanted material area 431. The first feature point P11 is a point indicating a corner of the document 41. The second feature point P12 is a point on the outer edge 413 of the document area 412 that touches the unwanted material area 431. The third feature point P13 is a point on the outer edge 413 that touches the outer edge 414 of the pre-processed image 400.

[0048] Step S204 is a determination process. The processing unit 112 that executes step S204 is an example of a "determination unit" in this disclosure. In step S204, the processing unit 112 selects one feature point P1 from among a plurality of feature points P1 as the "feature point of interest". The processing unit 112 further selects a feature point P1 adjacent to the feature point of interest on one side of the circumferential direction θ in the image coordinate system as a "neighboring feature point". The circumferential direction θ is the circumferential direction with respect to an axis that is orthogonal to the two coordinate axes that define the image coordinate system and passes through the reference point P2.

[0049] "Feature point of interest" and "neighboring feature point" are examples of "two adjacent feature points" in this disclosure.

[0050] Next, the processing unit 112 determines whether the feature point of interest and the neighboring feature points lie on the same side of the document 41. In detail, the processing unit 112 first determines whether the feature point of interest satisfies the first condition. The first condition is that the feature point of interest is the first feature point P11. If the first condition is met, it is determined that the feature point of interest and the neighboring feature points lie on the same side of the document 41.

[0051] If the feature point of interest does not satisfy the first condition, the processing unit 112 determines whether the feature point of interest and the neighboring feature points satisfy the second condition. The second condition is that each of the feature point of interest and the neighboring feature points is the second feature point P12. If the second condition is satisfied, it is determined that the feature point of interest and the neighboring feature points do not lie on the same edge in document 41.

[0052] If the second condition is not met, the processing unit 112 further determines whether the feature point of interest and the neighboring feature point satisfy the third condition. The third condition is that each of the feature point of interest and the neighboring feature point is the third feature point P13. If the third condition is met, it is determined that the feature point of interest and the neighboring feature point do not lie on the same side in document 41.

[0053] If the third condition is not met, the processing unit 112 further determines whether the feature point of interest and the neighboring feature point satisfy the fourth condition. The fourth condition is that the Euclidean distance between the feature point of interest and the neighboring feature point is less than or equal to the first threshold. If the fourth condition is met, it is determined that the feature point of interest and the neighboring feature point do not lie on the same side of document 41.

[0054] Note that the fourth condition may be changed to the fifth condition. The fifth condition is that the angle formed by sequentially connecting the feature point of interest, the reference point P2, and the neighboring feature points with line segments is less than or equal to the second threshold.

[0055] When the fifth condition is adopted, the accuracy of determining whether or not points are on the same side improves compared to the case of the fourth condition. Specifically, the Euclidean distance from document 41 to the imaging unit 12 can vary greatly depending on the installation position of the imaging unit 12. However, the angle formed by the feature point of interest, the reference point P2, and the neighboring feature points does not vary significantly depending on the installation position of the imaging unit 12. On the other hand, when the fourth condition is adopted, the determination of whether or not points are on the same side is simplified compared to the case of the fifth condition.

[0056] If none of the second, third, and fourth conditions are met, the processing unit 112 determines that the feature point of interest and the neighboring feature points lie on the same edge of the document 41. According to step S204, the feature points of interest and neighboring feature points on each edge of the document 41 are determined.

[0057] Step S205 is a corner identification process. The processing unit 112 that executes step S205 is an example of a “corner identification unit” in this disclosure. In step S205, the processing unit 112 identifies the four corners of document 41 based on the feature points of interest and neighboring feature points that are determined to be on the same side. In detail, the processing unit 112 selects one set from all combinations of feature points of interest and neighboring feature points determined in step S205 as the first feature point set. The processing unit 112 further selects a second feature point set from all combinations of feature points of interest and neighboring feature points. The second feature point set is a set of feature points of interest and neighboring feature points that are adjacent to the first feature point set on one side in the circumferential direction θ. The processing unit 112 determines one of the four corners of document 41 as the intersection point of the line passing through the feature points of interest and neighboring feature points in the first feature point set and the line passing through the feature points of interest and neighboring feature points in the second feature point set.

[0058] According to the identification process shown in Figure 7, multiple feature points P1 are determined in the reference point determination process (step S202). In the feature point identification process (step S203), it is determined whether the multiple feature points P1 are the first feature point P11, the second feature point P12, or the third feature point P13. In the determination process (step S204), it is determined, based on the first to fourth conditions, whether two adjacent feature points P1 among the multiple feature points P1 lie on the same side of the document 41. As a result, even if the entire main surface of the document 41 is not visible in the pre-processing image 400, the four corners of the document 41 are detected in the corner identification process (step S205).

[0059] Embodiments of the present disclosure have been described above with reference to the drawings. However, the present disclosure is not limited to the embodiments described above, and can be implemented in various forms without departing from its essence. Furthermore, the multiple components disclosed in the above embodiments can be modified as appropriate. For example, some components from all the components shown in one embodiment may be added to the components of another embodiment, or some components from all the components shown in one embodiment may be removed from the embodiment.

[0060] Furthermore, the drawings schematically show each component in order to facilitate understanding of the invention, and the thickness, length, number, spacing, etc. of each component shown may differ from the actual dimensions due to the convenience of drawing creation. Also, the configuration of each component shown in the above embodiments is merely an example and is not particularly limiting, and it goes without saying that various modifications are possible without substantially departing from the effects of the present invention.

[0061] (1) In this embodiment, the image processing device 11 processed image data acquired from the imaging unit 12 and performed the processing shown in Figures 5 and 7. However, the image processing device 11 is not limited to this and may also process image data stored in the storage medium 16 and perform the processing shown in Figures 5 and 7.

[0062] (2) In the embodiment, the image processing device 11 identified an unwanted object region 431 in which pixels representing hands as unwanted objects 43 are distributed by region identification processing (step S201) through skin-color region identification processing. However, unwanted objects 43 include not only hands but also various other objects such as writing instruments.

[0063] In order to identify the unwanted object region 431 where pixels representing various unwanted objects 43 are distributed, the processing unit 122 sends request data to the server device 300 in the region identification process (step S201). The request data includes the pre-processing image data acquired in step S101. In the server device 300, the processing unit 33 assigns labels to all pixels constituting the pre-processing image 400 indicated by the pre-processing image data through AI-based object recognition processing. The processing unit 33 sends response data including the label for each pixel to the information processing device 100.

[0064] In the image processing device 11, the processing unit 112 identifies pixels in the pre-processed image 400 that are included in the document area 412 and pixels that are included in the unwanted object area 431, based on each label included in the response data. Specifically, the unwanted object area 431 is identified for each of the multiple unwanted objects 43. AI-based object recognition processing makes it possible to identify the unwanted object area 431 for each of the various unwanted objects 43. For example, when the unwanted object area 431 is identified using a distance sensor, it is difficult to detect thin unwanted objects 43 such as pieces of paper or rulers. However, with AI-based object recognition processing, even thin unwanted objects 43 can be detected.

[0065] The AI-based object recognition processing may also be performed in the processing unit 112 of the image processing device 11.

[0066] (3) In the embodiment, the processing unit 112 performed a determination process (step S204) based on the feature point of interest and neighboring feature points adjacent to the feature point of interest on one side in the circumferential direction θ. However, in addition to this, the processing unit 112 may further perform a determination process based on the feature point of interest and neighboring feature points adjacent to the feature point of interest on the other side in the circumferential direction θ. Alternatively, the processing unit 112 may perform a determination process based on the feature point of interest and a plurality of neighboring feature points that are close to the feature point of interest on one side in the circumferential direction θ. [Industrial applicability]

[0067] This invention can be applied to computer programs for managing various types of documents (such as forms and business cards), and to computer devices capable of executing such programs. [Explanation of Symbols]

[0068] 100 Information Processing Devices 11 Image Processing Device 111 Storage section 112 Processing Unit 112A Image acquisition unit 112B Specific part 112C Deformed section 112D Character recognition section 112E Synthesis Department 113 Image Processing Programs 12 Imaging Unit 13 Display section 300 Server Devices 31 Communication Interface Section 32 Storage section 33 Processing Unit

Claims

1. A region identification unit that identifies a first region representing the document and a second region representing objects other than the document and the background from an image including the background, A feature point identification unit identifies a plurality of feature points based on the first region and the second region, including at least one of the following: a point that is a corner of the document, a point on the outer edge of the first region that touches the outer edge of the second region, and a point on the outer edge of the first region that touches the outer edge of the image. A determination unit that determines whether two adjacent feature points among the plurality of feature points lie on the same side in the document, Based on the two adjacent feature points determined to be on the same side, a corner identification unit identifies the four corners of the document. An image processing device equipped with the following features.

2. The system includes a reference point determination unit that determines a reference point based on the aforementioned plurality of feature points, The image processing apparatus according to claim 1, wherein the determination unit identifies two adjacent feature points from among the plurality of feature points based on the reference point, and determines whether the two adjacent feature points lie on the same side in the document.

3. The image processing apparatus according to claim 1 or 2, wherein the determination unit determines that the two adjacent feature points are on the same side if the distance between the two adjacent feature points is greater than or equal to a first threshold.

4. The image processing apparatus according to claim 2, wherein the determination unit determines that the two adjacent feature points are on the same side if the angle determined by one of the two adjacent feature points, the reference point, and the other of the two adjacent feature points is greater than or equal to a second threshold.

5. The image processing apparatus according to claim 1 or claim 2, comprising a deformation unit that generates a rectangular image obtained by deforming the first region into a rectangle based on the four corners of the document.

6. A character recognition unit performs character recognition processing on the first region, A synthesis unit that combines the recognition results of the character recognition unit with the rectangular image. The image processing apparatus according to claim 5, comprising:

7. The image processing apparatus according to claim 6, wherein the rectangular image synthesized by the synthesis unit is displayed on the display unit.

8. The image processing apparatus according to claim 1 or claim 2, wherein the aforementioned image is captured by an imaging device.

9. A computer identifies from an image including a background and a document a first region representing the document and a second region representing objects other than the document and the background, The computer identifies a plurality of feature points based on the first region and the second region, which are any of the following: a point that is a corner of the document, a point on the outer edge of the first region that touches the outer edge of the second region, and a point on the outer edge of the first region that touches the outer edge of the image. The computer determines whether two adjacent feature points among the plurality of feature points lie on the edge of the document, The computer determines the four corners of the document based on the two adjacent feature points that are determined to be on the edge. Image processing methods, including those mentioned above.

10. On the computer, A step of identifying a first region representing the document and a second region representing objects other than the document and the background from an image including the background, A step of identifying a plurality of feature points based on the first region and the second region, which are any of the following: a point that is a corner of the document, a point on the outer edge of the first region that touches the outer edge of the second region, and a point on the outer edge of the first region that touches the outer edge of the image. The step of determining whether two adjacent feature points among the plurality of feature points lie on the edge of the document, A step of identifying the four corners of the document based on the two adjacent feature points determined to be on the aforementioned edge, and A computer program that executes something.

Images