Image processing method and image processing device
By extracting and determining the feature points of the captured image, the problem of difficulty in determining the position at high shooting magnification is solved. The threshold determination and reference image mechanism are used to achieve high-precision determination of the position of the captured image in the music score or book.
Patent Information
- Application Number
- CN201980035307.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-05-30
- Filing Date
- 2019-05-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2039-05-24
AI Technical Summary
When the shooting device is sufficiently close to the musical score or the paper surface of the book, it is difficult to determine the position of the captured image with high accuracy, especially when the number of feature points is insufficient, it is impossible to properly compare the captured image with the candidate image, resulting in difficulty in determining the position.
By extracting the feature points of the captured image, determining whether the number of them exceeds the threshold. If it is insufficient, the position is determined relative to the previous captured image. The image with the number of feature points exceeding the threshold is used as a reference image for comparison to ensure the accuracy of the position.
At high shooting magnification, the location of the captured image in the music score or book can be appropriately determined, ensuring the accuracy and reliability of image processing, and avoiding position errors caused by insufficient number of feature points.
Smart Images

Figure CN112166456B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a technology for analyzing an image obtained by photographing written materials such as musical scores or books. Background Art
[0002] Conventionally, a technique for searching for music desired by a user has been proposed. For example, Patent Document 1 discloses a technique for searching for music including a time series of notes sequentially designated by a user from a plurality of candidates.
[0003] Patent Document 1: Japanese Patent Application Laid-Open No. 6-110945 Summary of the invention
[0004] By comparing an image obtained by photographing a music score consisting of multiple pages with a camera (hereinafter referred to as a "captured image") with a plurality of candidate images representing different pages in the music score, the position of the captured image in the music score (e.g., page number) can be determined. However, when the shooting magnification of the camera is sufficiently high (e.g., when the camera is sufficiently close to the paper surface of the music score), it will be difficult to properly compare the captured image and the candidate images. Therefore, it is difficult to estimate the position of the captured image in the music score with high accuracy. In addition, in the above description, an example of a captured image photographed of a music score is shown, but the same problem may also occur when analyzing an image obtained by photographing a document such as a novel. In view of the above situation, the purpose of the method of the present invention is to be able to properly determine the position of a captured image in a document such as a music score or a book.
[0005] In order to solve the above problems, the image processing method involved in the present invention is to extract feature points from captured images generated sequentially by a shooting device, determine whether the number of feature points extracted from the first captured image exceeds a threshold, and when the number of feature points extracted from the first captured image is lower than the threshold, determine the position of the first captured image relative to a second captured image generated previously compared with the first captured image.
[0006] The image processing device involved in the method of the present invention comprises: an extraction unit, which extracts feature points from captured images generated sequentially by a capturing device; a determination unit, which determines whether the number of feature points extracted from the first captured image exceeds a threshold value; and an analysis unit, which, when the number of feature points extracted from the first captured image is lower than the threshold value, determines the position of the first captured image relative to a second captured image generated previously compared with the first captured image. BRIEF DESCRIPTION OF THE DRAWINGS
[0007] Figure 1This is a block diagram showing the structure of the image processing device according to the first embodiment.
[0008] Figure 2 It is a schematic diagram illustrating the relationship between the imaging magnification of the imaging device and the range of the captured image.
[0009] Figure 3 This is an example of display by the display device when the number of feature points exceeds a threshold value.
[0010] Figure 4 This is an example of display by the display device when the number of feature points is lower than a threshold value.
[0011] Figure 5 This is a flowchart illustrating a specific procedure of processing executed by the control device.
[0012] Figure 6 This is an illustration of the staff that makes up a musical score.
[0013] Figure 7 This is a diagram to explain the analysis data.
[0014] Figure 8 This is a flowchart of the image analysis process in the second embodiment. DETAILED DESCRIPTION
[0015] <First embodiment>
[0016] Figure 1 1 is a block diagram illustrating the structure of the image processing device 100 according to the first embodiment. Figure 1 As shown in the example in the figure, the image processing device 100 of the first embodiment is realized by a computer system having a control device 11, a storage device 12, a camera 13, and a display device 14. For example, a mobile information terminal such as a mobile phone or a smart phone, or a mobile or fixed information terminal such as a personal computer can be used as the image processing device 100.
[0017] The photographing device 13 is an image input device that generates an image obtained by photographing a subject (hereinafter referred to as a "captured image"). Specifically, the photographing device 13 has a photographing element that photographs a subject within a photographing range corresponding to the field of view of an optical system such as a photographing lens. The captured image is represented by image data in any form. The photographing device 13 of the first embodiment generates captured images sequentially at a predetermined period. That is, a dynamic image composed of a time series of multiple captured images is generated. In addition, in the first embodiment, a structure in which the photographing device 13 is mounted on the image processing device 100 is illustrated, but the photographing device 13, which is separate from the image processing device 100, can also be connected to the image processing device 100 by wire or wirelessly. That is, the photographing device 13 can also be omitted from the image processing device 100.
[0018] The user uses the camera 13 to shoot the music score of any music piece. The user can move the camera 13 closer to or farther from the sheet of music score as desired. Figure 2 As shown in the example, the range represented by the captured image in the music score changes over time. Figure 2 In the figure, the time series of captured images sequentially generated by the camera 13 in the process of gradually approaching the paper surface of the music score are arranged from the left to the right of the figure. Figure 2 As shown in the example, the closer the camera 13 is to the sheet of the score, the greater the shooting magnification is and the smaller the shooting range in the score is. In addition, the shooting magnification of the camera 13 can also be optically adjusted by operating an optical system such as a shooting lens. In addition, the shooting magnification can also be electronically adjusted by image processing of image data of the captured image.
[0019] The control device 11 is, for example, a processing circuit such as a CPU (Central Processing Unit), and centrally controls the various elements of the image processing device 100. The control device 11 of the first embodiment determines the score information and the page number by analyzing the captured images sequentially generated by the imaging device 13. The score information is information for identifying the score represented by the captured image. For example, the name of the music represented by the score, or the identification number uniquely assigned to the score is used as the score information. The page number is the number of the page represented by the captured image among the multiple pages constituting the score. That is, the page number indicates the position of the portion represented by the captured image in the score.
[0020] The storage device 12 stores programs executed by the control device 11 and various data used by the control device 11. As the storage device 12, for example, a known recording medium such as a semiconductor recording medium or a magnetic recording medium, or a combination of a plurality of recording media can be arbitrarily adopted.
[0021] The storage device 12 of the first embodiment stores reference data R for each of a plurality of images (hereinafter referred to as "candidate images") prepared in advance. Each of the plurality of candidate images is an image corresponding to one page of a musical score whose musical score information and page number are known. The plurality of reference data R corresponding to different candidate images are stored in the storage device 12 for each of a plurality of musical pieces. The reference data R corresponding to any one of the candidate images includes image data C representing the candidate image and attached data D related to the candidate image. The attached data D of each candidate image specifies the musical score information and page number of the musical score represented by the candidate image.
[0022] The display device 14 displays various images based on the control of the control device 11. For example, a liquid crystal display panel or an organic EL (Electroluminescence) display panel is preferably used as the display device 14. Figure 3 As shown in the example, the captured images sequentially generated by the camera 13 are displayed in real time in parallel with the shooting by the camera 13. That is, a live view display of the captured images is performed. In addition, the display device 14 displays the information area Q together with the captured images. The information area Q includes the score information (Book) of the score represented by the captured image and the page number (Page) corresponding to the captured image.
[0023] The music score information and page number indicated by the attached data D of one candidate image corresponding to the photographed image among the plurality of candidate images for which the reference data R is stored in the storage device 12 are displayed in the information area Q. The candidate image corresponding to the photographed image is, for example, a candidate image including a portion similar to the photographed image. The user can grasp the music score information and page number of the music score currently being photographed by the photographing device 13 by visually recognizing the information area Q of the display device 14.
[0024] like Figure 1 As shown in the example in the figure, the control device 11 of the first embodiment realizes a plurality of functions (extraction unit 21, determination unit 22, determination unit 23, analysis unit 24, display control unit 25) for analyzing the captured images sequentially generated by the imaging device 13. In addition, the functions of the control device 11 may be realized by a collection of a plurality of devices (i.e., a system), or a part or all of the functions of the control device 11 may be realized by a dedicated electronic circuit.
[0025] The extraction unit 21 extracts a plurality of feature points from the captured images sequentially generated by the imaging device 13. Feature points are characteristic locations within the captured images. The extraction of feature points is performed sequentially for each captured image. When extracting feature points, a well-known image analysis technique such as SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), KAZE (KAZE Features), or AKAZE (Accelerated KAZE) is arbitrarily used. The number N of feature points extracted from each captured image is different for each captured image. Specifically, Figure 2 As shown in FIG. 2 , there is a tendency that the number N of feature points decreases as the imaging magnification of the imaging device 13 increases (that is, the imaging range in the musical score decreases).
[0026] The determination unit 22 determines a candidate image corresponding to the captured image from among a plurality of candidate images. Specifically, the determination unit 22 calculates a similarity index by comparing the candidate image with the captured image for each of the plurality of candidate images, and determines a candidate image having the largest similarity index from among the plurality of candidate images. The similarity index is an index indicating the degree of similarity between the candidate image and the captured image. For example, a similarity index indicating the degree of similarity between the distribution of feature points of the candidate image and the distribution of feature points of the captured image is calculated. A known image processing technique is arbitrarily used when comparing the candidate image with the captured image. By determining a candidate image corresponding to the captured image (for example, a candidate image including a portion similar to the captured image), the score information and page number specified by the attached data D of the candidate image are determined as the score information and page number of the captured image. As understood from the above description, the determination unit 22 functions as an element for determining the score information and page number of the captured image.
[0027] As described above, the more the shooting magnification of the shooting device 13 increases, the more the number N of feature points decreases. When the number N of feature points is small, it is difficult to properly compare the captured image with each candidate image. In other words, it is difficult to calculate a similarity index that properly expresses the degree of similarity between the captured image and the candidate images. Considering the above situation, in the first embodiment, the presence or absence of comparison between the captured image and each candidate image is controlled in accordance with the number N of feature points in the captured image.
[0028] Figure 1The determination unit 23 determines whether the number N of feature points extracted from the captured image exceeds a predetermined threshold value Nth. The threshold value Nth is set by statistics or experiments in such a way that the number of feature points of the candidate image that can be properly compared with the candidate image exceeds the minimum value. When the number N of feature points of the captured image exceeds the threshold value Nth, the determination unit 22 performs a comparison between the captured image and each candidate image. On the other hand, when the number N of feature points is lower than the threshold value Nth, the determination unit 22 does not perform a comparison between the captured image and each candidate image. That is, the score information and page number of the captured image are not determined.
[0029] When the number N of feature points extracted from a captured image is less than a threshold value Nth, the analyzing unit 24 determines the position of the captured image relative to a specific reference image. The reference image is, for example, a captured image generated by the imaging device 13 before (in the past) the captured image (an example of a second captured image). Specifically, the most recent (i.e., the newest) captured image in which the number N of feature points exceeds the threshold value Nth is used as the reference image. Figure 2 As shown in the example, as the shooting magnification increases (i.e., the score is enlarged), the number of feature points N decreases, so the captured image with the number of feature points N less than the threshold value Nth corresponds to a part of the reference image. The analyzing unit 24 of the first embodiment generates position information indicating the position of the captured image relative to the reference image. The position information is information indicating the relative position of the captured image relative to the reference image, and is expressed, for example, by coordinates in a coordinate plane having two axes orthogonal to each other.
[0030] The display control unit 25 causes the display device 14 to display various images. The display control unit 25 of the first embodiment displays the captured images sequentially generated by the imaging device 13 and the information area Q related to the captured images on the display device 14 in parallel with the images taken by the imaging device 13. Specifically, regarding the captured images whose number of feature points N exceeds the threshold value Nth, as shown in FIG. Figure 3 As shown in the example of FIG. 1 , the reference information including the score information and the page number determined by the determination unit 22 through comparison with each candidate image is displayed on the display device 14. On the other hand, for a captured image in which the number N of feature points is less than the threshold value Nth, as shown in FIG. Figure 4 As shown in the example in FIG. 1 , the music score information and page number specified for the reference image and the position information (specifically, the X coordinate and the Y coordinate) generated for the captured image by the analyzing unit 24 are displayed. Therefore, even when the capturing magnification of the image capturing device 13 is sufficiently high (for example, when the image capturing device 13 is sufficiently close to the sheet of the music score), the user can understand the position of the captured image.
[0031] Figure 51 is a flowchart illustrating a specific procedure of processing (an example of an image processing method) executed by the image processing device 100. Figure 5 If you start Figure 5 The extraction unit 21 extracts feature points from the captured image (example of the first captured image) (Sa1). The determination unit 23 determines whether the number N of feature points exceeds the threshold value Nth (Sa2).
[0032] When the determination unit 23 determines that the number N exceeds the threshold value Nth (Sa2: YES), the determination unit 22 determines the candidate image corresponding to the captured image among the plurality of candidate images (Sa3). Specifically, the determination unit 22 calculates a similarity index for each candidate image by comparing the plurality of candidate images with the captured image, and determines the candidate image with the largest similarity index among the plurality of candidate images. The display control unit 25 displays the score information and page number specified by the attached data D of the candidate image corresponding to the captured image as the score information and page number of the captured image in the information area Q (Sa4).
[0033] In addition, when the number N exceeds the threshold value Nth, the determination unit 22 updates the reference image stored in the storage device 12 to the current captured image (Sa5). That is, the reference image is updated each time a captured image in which the number N of feature points exceeds the threshold value Nth is captured. As understood from the above description, the latest captured image in which the number N of feature points exceeds the threshold value Nth is stored as the reference image.
[0034] On the other hand, when the determination unit 23 determines that the number N is less than the threshold value Nth (Sa2: NO), the analysis unit 24 determines the position of the captured image relative to the reference image stored in the storage device 12 (Sa6). That is, position information indicating the position of the captured image relative to the reference image is generated. The display control unit 25 displays the information area Q including the score information and page number determined relative to the reference image and the position information of the captured image at the current time on the display device 14 (Sa7). In addition, when the number N is equal to the threshold value Nth, either the search for candidate images (Sa3) or the generation of position information (Sa6) may be performed.
[0035] As described above, in the first embodiment, when the number N of feature points extracted from the captured image exceeds the threshold value Nth, that is, when the captured image and each candidate image can be appropriately compared, the candidate image corresponding to the captured image among the plurality of candidate images is determined. On the other hand, when the number N of feature points extracted from the captured image is lower than the threshold value Nth, that is, when the captured image and each candidate image cannot be appropriately compared due to the sufficiently high shooting magnification of the shooting device 13, the position of the captured image relative to the reference image generated before the captured image is determined. Therefore, the user can appropriately grasp the position of the captured image.
[0036] In addition, a captured image in which the number of feature points N exceeds the threshold value Nth, that is, a past captured image that can be appropriately compared with each candidate image, is used as a reference image, and the position of the captured image relative to the reference image is determined. Therefore, there is an advantage that the user can appropriately grasp the position of the captured image in the music score. In the first embodiment, in particular, the most recent captured image in which the number of feature points N exceeds the threshold value Nth is used as the reference image. Therefore, compared with a structure in which the time when the captured image is generated and the time when the reference image is generated are separated on the time axis, there is an advantage that the position of the captured image in the music score can be appropriately determined.
[0037] <Second embodiment>
[0038] In the following examples, the same reference numerals as those in the first embodiment are used for elements having the same functions as those in the first embodiment, and detailed descriptions thereof are appropriately omitted.
[0039] The music score photographed by the photographing device 13 consists of multiple pages. Figure 6 As shown in the example in FIG. 1 , each page of the musical score includes a plurality of staves F arranged in the vertical direction at intervals from each other. Each of the plurality of staves F is composed of a plurality of horizontal straight lines representing different pitches. Specifically, each stave F is a five-line stave composed of five straight lines. A plurality of musical score elements including part marks such as treble marks or tenor marks and marks such as notes or rests are arranged with respect to the plurality of staves F, respectively.
[0040] Figure 7 FIG. 1 is a schematic diagram for explaining the image data C of the candidate image in the second embodiment. Figure 7 The middle figure shows a plurality of feature points Pa extracted from a candidate image and a reference line Lref along a plurality of straight lines L constituting a spectrum F of the candidate image. Figure 7The projection point Pb is the location where one feature point Pa is projected (orthogonally projected) onto the baseline Lref. That is, the projection point Pb is the intersection (foot of the perpendicular) between the perpendicular line to the baseline Lref passing through one feature point Pa and the baseline Lref. The image data C of each candidate image in the second embodiment includes a plurality of analysis data Z corresponding to different spectra F of the candidate image. The analysis data Z corresponding to any one spectra F in the candidate image is vector data representing a set of a plurality of projection points Pb that respectively project a plurality of feature points Pa of the candidate image relative to the baseline Lref corresponding to the spectra F.
[0041] The determination unit 22 of the second embodiment generates analysis data Z by analyzing the captured image captured by the imaging device 13, and compares the multiple analysis data Z stored in the storage device 12 with the analysis data Z of the captured image for different candidate images, thereby determining the candidate image corresponding to the captured image.
[0042] Figure 8 1 is a flowchart illustrating a specific procedure of a process (hereinafter referred to as “image analysis process”) in which the control device 11 of the second embodiment determines a candidate image corresponding to a captured image. Figure 5 Step Sa3 is executed Figure 8 When the image analysis process is started, the specifying unit 22 generates analysis data Z (Sb1 to Sb4) by analyzing the captured image.
[0043] The determination unit 22 sets a reference line Lref for the captured image (Sb1). Specifically, the determination unit 22 extracts the straight lines L constituting the stave F from the captured image, and sets a reference line Lref parallel to the plurality of straight lines L. For example, the determination unit 22 divides (clusters) the plurality of horizontal straight lines L extracted from the captured image for each stave F, and sets a reference line Lref that passes through the midpoint of the stave F in the vertical direction and extends horizontally along the straight lines L of the stave F.
[0044] The determination unit 22 divides the plurality of feature points Pa extracted from the captured image in step Sa1 into a plurality of sets (clusters) for each spectrum F (Sb2). A known clustering technique is arbitrarily used when dividing the plurality of feature points Pa. In addition, when only one spectrum F is included in the captured image, the processing of step Sb2 can be omitted.
[0045] The determination unit 22 determines a plurality of projection points Pb for each spectrum F included in the captured image by projecting a plurality of feature points Pa onto the reference line Lref (Sb3). That is, the determination unit 22 orthogonally projects a plurality of feature points Pa extracted from the vicinity of an arbitrary spectrum F onto the reference line Lref, thereby determining a plurality of projection points Pb. The determination unit 22 generates analysis data Z (Sb4) representing the plurality of projection points Pb determined by the above-described processing. In addition, the analysis data Z of each candidate image is generated by executing the above-described exemplified processing (Sb1 to Sb4) on the candidate image and is stored in the storage device 12.
[0046] When the analysis data Z of the captured image is generated through the above processing, the determination unit 22 compares the plurality of analysis data Z stored in the storage device 12 for different candidate images with the analysis data Z of the captured image, thereby determining the candidate image corresponding to the captured image (Sc1 to Sc4). The specific processing sequence is as follows.
[0047] The determination unit 22 selects any one of the plurality of candidate images (Sc1), and calculates a similarity index between the analysis data Z of the candidate image and the analysis data Z of the captured image (Sc2). For example, any index indicating the distance or correlation between the analysis data Z of the candidate image and the analysis data Z of the captured image is suitable as the similarity index. In the following description, it is assumed that the more similar the analysis data Z of the candidate image and the analysis data Z of the captured image are, the larger the value of the similarity index becomes.
[0048] The determination unit 22 determines whether the similarity index between all candidate images and the captured image has been calculated (Sc3). If there are candidate images for which the similarity index has not been calculated (Sc3: NO), the determination unit 22 reselects the unselected candidate images and calculates the similarity index between the analysis data Z of the candidate images and the analysis data Z of the captured image (Sc2).
[0049] If similarity indices are calculated for all candidate images by repeatedly performing the above-exemplified processing (Sc3: YES), the determination unit 22 determines the candidate image with the largest similarity index among the plurality of candidate images (Sc4). That is, the candidate image corresponding to the captured image among the plurality of candidate images is determined. The specific order of the image analysis processing is as described above.
[0050] The same effect as that of the first embodiment is also achieved in the second embodiment. In the second embodiment, the analysis data Z representing the result of projecting the plurality of feature points Pa extracted from the captured image onto the baseline Lref and the plurality of analysis data Z corresponding to different candidate images are respectively compared. Therefore, the candidate image corresponding to the captured image can be determined with high precision. In addition, the baseline Lref corresponding to the straight line L contained in the captured image is set, so there is also an advantage that, for example, the user does not need to set the baseline Lref. However, the user can also specify the baseline Lref. In addition, the method for determining the candidate image corresponding to the captured image is not limited to the image analysis processing of the second embodiment.
[0051] <Modification>
[0052] Specific variations of the above-described embodiments are exemplified below. Two or more embodiments arbitrarily selected from the following embodiments may be combined as appropriate within the scope of non-contradiction.
[0053] (1) In each of the above-mentioned methods, the most recent captured image in which the number of feature points N exceeds the threshold value Nth is used as the reference image, but the reference image is not limited to the above examples. For example, a captured image captured by the imaging device 13 at the time when the user performs a predetermined operation on the input device (not shown) of the image processing device 100 may be used as the reference image.
[0054] (2) For example, the image processing device 100 may be implemented by a server device that communicates with a terminal device (e.g., a mobile phone or a smart phone) via a communication network such as the Internet. For example, the image processing device 100 receives a captured image from the terminal device and transmits position information determined by analyzing the captured image to the terminal device.
[0055] (3) In each of the above-mentioned methods, the case where the photographing device 13 photographs a musical score is exemplified, but the subject represented by the photographed image is not limited to the musical score. For example, the present invention can also be applied to determine a candidate image (for example, a corresponding amount of one page of the document) from a photographed image obtained by photographing a document such as a book. As understood from the above description, one method of the present invention is an image processing method for analyzing a photographed image obtained by photographing a planar subject (for example, a musical score or a document) to determine a candidate image corresponding to the photographed image from a plurality of candidate images.
[0056] (4) The image processing device 100 involved in each of the aforementioned methods is implemented by the collaborative action of the control device 11 and the program, as illustrated in each method. The program involved in each of the aforementioned methods can be provided and installed in a computer by being stored in a computer-readable recording medium. The recording medium is, for example, a non-volatile (non-transitory) recording medium, preferably an optical recording medium (optical disc) such as a CD-ROM, but may also include any known form of recording medium such as a semiconductor recording medium or a magnetic recording medium. In addition, the non-volatile recording medium includes any recording medium other than a temporary transmission signal (transitory, propagating signal), and does not exclude volatile recording media. In addition, the program can also be provided to the computer by transmission via a communication network.
[0057] (5) According to the above-exemplified embodiment, for example, the following structure can be understood.
[0058] The image processing method involved in the mode (first mode) of the present invention extracts feature points from captured images sequentially generated by a camera, determines whether the number of feature points extracted from the first captured image exceeds a threshold, and when the number of feature points extracted from the first captured image is lower than the threshold, determines the position of the first captured image relative to the second captured image generated earlier than the first captured image. In the above mode, when the number of feature points extracted from the first captured image is lower than the threshold, that is, when the first captured image and each candidate image cannot be properly compared because the camera is sufficiently close to the subject, the position of the first captured image relative to the second captured image generated earlier than the first captured image is determined. Therefore, the position of the captured image in a document such as a musical score can be properly determined.
[0059] In the first mode, when the number of feature points extracted from the first captured image exceeds the threshold, the candidate image corresponding to the first captured image among the plurality of candidate images can be determined. In the above mode, when the number of feature points extracted from the first captured image exceeds the threshold, that is, when the first captured image and each candidate image can be appropriately compared, for example, the candidate image corresponding to the first captured image among the plurality of candidate images can be determined. Therefore, the position of the captured image within a document such as a musical score can be appropriately determined.
[0060] In the first mode, the second captured image may be a captured image in which the number of feature points exceeds the threshold. In the above mode, the position of the first captured image relative to the second captured image in which the number of feature points exceeding the threshold is extracted, that is, relative to a past captured image that can be appropriately compared with the candidate image is determined. Therefore, the position of the captured image in a document such as a musical score can be appropriately determined.
[0061] In the first mode, the second captured image may be a captured image immediately before the first captured image, in which the number of feature points exceeds the threshold. In the above mode, a past captured image immediately before the first captured image, in which the number of feature points exceeding the threshold is extracted, is used as the second captured image, and the position of the first captured image relative to the second captured image is determined. Therefore, compared with a structure in which the time when the first captured image is generated and the time when the second captured image is generated are deviated on the time axis, the position of the captured image in a document such as a musical score can be appropriately determined.
[0062] The embodiments of the present invention can also be realized as an image processing device that executes the image processing method of each embodiment described above, or as a program that causes a computer to execute the image processing method of each embodiment described above.
[0063] This application is based on Japanese patent application No. 2018-103440 filed on May 30, 2018, the contents of which are incorporated herein by reference.
[0064] Description of the label
[0065] 100…image processing device, 11…control device, 12…storage device, 13…shooting device, 14…display device, 21…extraction unit, 22…determination unit, 23…judgment unit, 24…analysis unit, 25…display control unit, F…spectrum, Lref…baseline, Pa…feature point, Pb…projection point.
Claims
1. An image processing method, which is implemented by a computer, extracting feature points from captured images sequentially generated by capturing any one page of a subject consisting of a plurality of pages by a capturing device, the captured images including at least a first captured image and a second captured image generated earlier than the first captured image, determining whether the number of feature points extracted from the first captured image exceeds a threshold, When the number of feature points extracted from the first captured image is less than the threshold, determining the relative position of the first captured image with respect to the second captured image in the subject, The second captured image is a captured image in which the number of the feature points exceeds the threshold value, The first captured image is an image showing a portion of the second captured image.
2. The image processing method according to claim 1, wherein: When the number of feature points extracted from the first captured image exceeds the threshold, a candidate image corresponding to the first captured image among a plurality of candidate images is identified.
3. The image processing method according to claim 1, wherein: The second captured image is a captured image immediately before the first captured image and in which the number of feature points exceeds the threshold.
4. The image processing method according to claim 2, wherein: The candidate image corresponding to the first captured image is determined by comparing analytical data including projection points obtained by projecting the feature points in the first captured image onto a baseline and analytical data including projection points obtained by projecting each feature point in the plurality of candidate images onto a baseline.
5. The image processing method according to claim 1 or 2, wherein: Position information indicating the position of the first captured image relative to the second captured image is displayed on a display device.
6. An image processing device, comprising: an extracting unit that extracts feature points from captured images that are sequentially generated by a camera capturing any one of a plurality of pages of a subject, the captured images including at least a first captured image and a second captured image that is generated earlier than the first captured image; a determination unit configured to determine whether the number of feature points extracted from the first captured image exceeds a threshold; as well as an analyzing unit that determines a relative position of the first captured image with respect to the second captured image on the subject when the number of feature points extracted from the first captured image is less than the threshold value, The second captured image is a captured image in which the number of the feature points exceeds the threshold value, The first captured image is an image showing a portion of the second captured image.
7. The image processing device according to claim 6, wherein: The method further includes a specifying unit that specifies a candidate image corresponding to the first captured image among a plurality of candidate images when the number of feature points extracted from the first captured image exceeds the threshold value.
8. The image processing device according to claim 6, wherein: The second captured image is a captured image immediately before the first captured image and in which the number of feature points exceeds the threshold.
9. The image processing apparatus according to claim 7, wherein: The determination unit determines the candidate image corresponding to the first captured image by comparing analytical data including projection points obtained by projecting the feature points in the first captured image onto a baseline and analytical data including projection points obtained by projecting each feature point in the multiple candidate images onto a baseline.
10. The image processing device according to claim 6 or 7, wherein: A display device is further provided, the display device displaying position information indicating the position of the first captured image relative to the second captured image.
Citation Information
Patent Citations
Music data base preparing device and retrieving device for the same
JP1994110945A
Image recording apparatus for recording and shooting image and image recording method
CN102685362A
Image retrieval device and method, and image processing device and method
CN103329166A
Image processing device and image processing method
CN104412303A
Image composition device, image composition method, and program
JP2014123916A
