Frame extraction method and program

Abstract

To provide a frame extraction method that can extract frames from a cartoon image simply and accurately.SOLUTION: A frame extraction method includes: a binarization step of binarizing a cartoon image into black and white; a first extraction step of extracting an outline of the cartoon image and extracting areas surrounded by frames in the cartoon image as frames; a clustering step of clustering black areas with respect to a remaining area not extracted as the frames by the first extraction step in the cartoon image; and a second extraction step of setting each of the clustered areas as a rectangular area and extracting the rectangular area as a frame.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a frame extraction method and a program. [Background technology]

[0002] In recent years, the opportunities to read manga on mobile devices such as smartphones have become increasingly common. However, manga is often more difficult to read on mobile devices such as smartphones than when it is printed on paper. For example, one page of a manga has multiple frames, and when one page is displayed as a single image, each frame becomes small and difficult to read. Also, when enlarging a frame on a page to read it, the operation becomes cumbersome and difficult to read.

[0003] In response to this, As in Patent Document 1 It is thought that by extracting frames from manga images and displaying each frame separately, it will be possible to make the images easier to read on mobile devices. 。

[0004] [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2014-212476 A Summary of the Invention [Problem to be solved by the invention]

[0006]

[0007] Therefore, the present invention in teeth, How to make manga easier to read on mobile devices The purpose is to provide. [Means for solving the problem]

[0008] In order to solve the above problems, the present invention The method executed by the computer includes a first extraction step of extracting contours of a comic image and extracting areas surrounded by frame lines in the comic image as frames, and a rearrangement step of rearranging the frames based on the positions of the frames in the comic image, and a new comic image having the rearranged frames is presented on a mobile terminal. It is characterized by: Effect of the Invention

[0009] According to the present invention, Making manga easier to read on mobile devices It is possible. [Brief description of the drawings]

[0010] [Figure 1] 1 is a flowchart of an example of a frame extraction method according to the present invention. [Diagram 2] 1A to 1C are diagrams for explaining a comic image in an example of a frame extraction method according to the present invention. [Diagram 3] FIG. 13 is a diagram for explaining a clustering process. [Figure 4] 13A to 13C are diagrams for explaining another comic image in an example of the frame extraction method according to the present invention. [Diagram 5] 13 is a flowchart of another example of a frame extraction method according to the present invention. [Figure 6] FIG. 13 is a diagram for explaining a rearrangement process. [Figure 7] FIG. 11 is another diagram for explaining the rearrangement process. [Figure 8] FIG. 11 is another diagram for explaining the rearrangement process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] The frame extraction method and program according to the present invention will be described below with reference to the drawings. Note that the present invention is not limited to the following embodiments, and may be modified, added, modified, deleted, or otherwise altered within the scope of what a person skilled in the art can conceive, and any embodiment is within the scope of the present invention as long as it provides the functions and effects of the present invention.

[0012] (First embodiment) An embodiment of a frame extraction method according to the present invention will be described. The frame extraction method of this embodiment is characterized by comprising a binarization process for binarizing a manga image into black and white, a first extraction process for extracting contours of the manga image and extracting areas surrounded by frame lines in the manga image as frames, a clustering process for clustering black areas from the remaining areas in the manga image that were not extracted as frames by the first extraction process, and a second extraction process for treating each of the clustered areas as a rectangular area and extracting the rectangular areas as frames.

[0013] According to this embodiment, frames can be extracted easily and accurately from a manga image. According to this embodiment, in addition to frames without borders, frames surrounded by wavy lines rather than straight lines, such as those drawn by hand, can be extracted. Here, "extracting frames with high accuracy" refers to extracting what a human would recognize as a frame, and is closer to human recognition than conventional techniques. It is preferable for the results to match human recognition as closely as possible, but perfect match is not required.

[0014] In addition, in this embodiment, frame extraction can be performed using machine learning technology based on unsupervised learning, and frame extraction can be performed more easily than with conventional technology. In conventional technology, it is necessary to prepare teacher data, and when performing machine learning based on supervised learning with a correct answer, it requires time and cost. In contrast, according to this embodiment, frame extraction can be performed using machine learning technology without preparing teacher data, and frame extraction can be performed easily and accurately. In addition, by combining machine learning without teacher data and region extraction, frame extraction can be performed accurately even for frames that do not have frame frames.

[0015] FIG. 1 shows a flowchart of the frame extraction method of this embodiment. In this embodiment, a binarization step (S101), a first extraction step (S102), a clustering step (S103), and a second extraction step (S104) are performed, and other steps are performed as necessary.

[0016] <Binarization process> In the binarization step, the cartoon image is binarized into black and white (S101). Although there is no particular limitation on the binarization, for example, a cartoon image is binarized by converting it into the HSV color space. When converting it into the HSV color space and binarizing it, for example, areas with a brightness of 80% or less and a hue of 70% or less are colored black, and the rest are colored white. It should be noted that the binarization step is not limited to the HSV color space, and other color spaces may be used for binarization.

[0017] In this embodiment, it is preferable to increase the accuracy of binarization in order to improve the accuracy of frame extraction in the first extraction process described below. In this embodiment, one way to increase the accuracy of binarization is to complement the binarization by grayscale using edge detection, for example.

[0018] <First extraction step> In the first extraction step, contour extraction is performed on the comic image, and areas enclosed by frame lines in the comic image are extracted as frames (S102).

[0019] The first extraction step of this embodiment will be described with reference to Fig. 2. Fig. 2(A) shows an example of a cartoon image, in which frames A to E are drawn in cartoon image 10. When contour extraction is performed on cartoon image 10, frames A to C and E are extracted, as shown in Fig. 2(B).

[0020] In contour extraction, an area surrounded by a frame line in a target image is recognized, and the area surrounded by the frame line is cut out and output. Therefore, when contour extraction is performed on comic image 10, frames A to C and E can each be extracted and output.

[0021] In the first extraction step of this example, frame D in comic image 10 is not subject to extraction. Frame D is not recognized as an area surrounded by a frame line in the contour extraction because the frame line is broken at a point. Frame D is extracted through the clustering and second extraction steps described below.

[0022] It should be noted that the frame line of frame E is a wavy line drawn by hand, but in this example it can be extracted as a frame. Even if frame E is not extracted, it can become an extraction target by going through the clustering and second extraction steps described below.

[0023] The contour extraction can be performed using a region growing method (watershed algorithm), a region merging method, or the like.

[0024] <Clustering process> In the clustering step, black regions are clustered for the remaining regions in the comic image that have not been extracted as the frames in the first extraction step (S103).

[0025] By performing clustering on the remaining areas that were not extracted as frames in the first extraction process, it is possible to extract frames that were not extracted in the first extraction process. In reality, frames are extracted in the second extraction process described below, but the clustering process corresponds to making preparations for that.

[0026] The clustering step of this embodiment will be described with reference to an example in FIG. Fig. 3(A) shows frame D in Fig. 2, and Fig. 3(B) shows an enlarged view of frame 20 in frame D. Fig. 3(B) also shows the state after clustering has been performed, and by performing clustering, even if there are discontinuous parts in frame 20, they can be treated as one group (clustering region 30).

[0027] As a result, as shown in Figure 3(C), the broken frame line 20 of frame D can be treated as a clustering region 30. If there is no continuous frame line, such as when the frame line is broken, the small regions will be detected separately, so by performing clustering, small regions that are close to each other are combined.

[0028] In the clustering process of this embodiment, clustering is performed on various regions, not limited to the frame lines in the above example, to create clusters. By creating clusters, rectangular regions for extracting frames can be formed in the second extraction process described below.

[0029] The clustering is preferably performed by density based spatial clustering of applications with noise (DBSCAN). DBSCAN has the advantage of being able to perform clustering with high accuracy, and frame extraction can be performed with high accuracy in the second frame extraction.

[0030] Parameters for clustering include, for example, a parameter that indicates how far apart dots can be before they are treated as a cluster, and a parameter that indicates how many dots must be around a particular dot before they are treated as a cluster. For example, when using DBSCAN, parameters include ε and minPts. DBSCAN is an unsupervised learning method, so there is no need to create supervised data, which reduces effort and costs.

[0031] <Second extraction step> In the second extraction step, each of the clustered regions is made into a rectangular region, and the rectangular region is extracted as a frame (S104).

[0032] As shown in FIG. 3C, since the clustering region 30 of frame D is a rectangular region, frame D is extracted as a frame by the second extraction step.

[0033] In the second extraction step, "each region" refers to the cluster described above, and the clustered region is defined as a rectangular region. The method of creating a rectangular region can be changed as appropriate. For example, four points are selected from a cluster, and the four points are arranged in a rectangular shape, or multiple straight lines are selected from a cluster, and the straight lines are arranged in a rectangular shape.

[0034] In this way, in the second extraction step in this embodiment, for example, rectangular areas are created from each clustered area, and the created rectangular areas can be extracted as frames. This allows frame extraction to be performed with greater accuracy than with conventional techniques.

[0035] In this embodiment, the second extraction step is performed after clustering the black areas of the remaining areas that were not extracted as frames by the first extraction step, but the second extraction step may also be performed after clustering the binarized manga image.

[0036] <Other examples> Here, other examples of this embodiment will be described. According to this embodiment, frame extraction can be performed even in manga images that do not have frame borders. An example is shown in Fig. 4. Fig. 4(A) is a diagram showing an example of a manga image, and Fig. 4(B) is a diagram showing frames from which frames have been extracted. According to this embodiment, frame extraction can be performed easily and accurately not only for manga images such as those shown in Fig. 2, but also for manga images that do not have frame borders such as those shown in Fig. 4.

[0037] <Program> The program of this embodiment is characterized in that it causes a computer to execute the following processes: a binarization process that converts a manga image into black and white; a first extraction process that performs contour extraction on the manga image and extracts areas surrounded by frame lines in the manga image as frames; a clustering process that clusters black areas in the remaining areas of the manga image that were not extracted as frames by the first extraction process; and a second extraction process that determines rectangular areas in the clustered manga image and extracts the rectangular areas as frames.

[0038] Second Embodiment Another embodiment of the frame extraction method according to the present invention will now be described. A flowchart of the frame extraction method of this embodiment is shown in Fig. 5. Note that the binarization step (S202), the first extraction step (S205), the clustering step (S206), and the second extraction step (S207) in this embodiment can be the same steps as those in the above embodiment, so their explanation will be omitted.

[0039] <Normalization process> In the normalization step, the horizontal size of the cartoon image is normalized before the binarization step (S201). By performing normalization, the accuracy of the first extraction step and the second extraction step can be improved. The horizontal size of the cartoon image can be changed as appropriate, but can be set to, for example, 300 pixels to 500 pixels.

[0040] Normalization is not essential and can be omitted, but in that case it is preferable that the width of the comic image is a predetermined size.

[0041] <Straight line determination process> In the straight line determination step, which is after the binarization step and before the first extraction step, a Hough transform is performed on the cartoon image to determine whether a straight line is detected (S203).

[0042] It is determined whether a straight line is detected (S205), and if it is determined that a straight line is detected in the straight line detection step (YES in S205), the first extraction step (S205), the clustering step (S206), and the second extraction step (S207) are performed. This makes it possible to prevent unnecessary steps from being performed and to speed up the processing.

[0043] There are a considerable number of single-frame images in manga images that do not even have a frame. Single-frame images often do not have straight lines, so if a manga image does not have straight lines, it is determined to be a single-frame image, and the first extraction step and subsequent steps are not carried out, and the image is output as a single frame (S209). This makes it possible to omit steps and thus save the time required for frame extraction.

[0044] A commonly used method can be used to detect lines using the Hough transform. For example, a line passing through a certain dot is ρ=x·cosθ+y·sinθ It is possible to draw as many curves on the ρ and θ coordinate axes as there are dots, and the intersections of the curves form straight lines passing through multiple dots.

[0045] In the line determination process, if a horizontal line whose length is 50% or more of the width of the comic image is detected after the Hough transform, it can be determined that the comic image contains a line. In addition, if normalization is performed, if a horizontal line whose length is 50% or more of the width of the normalized comic image is detected, it can be determined that the comic image contains a line.

[0046] <Sorting process> In the rearrangement step, the frames extracted in the first extraction step and the second extraction step are rearranged in accordance with the order in which the frames in the comic image are read (S208). By performing the rearrangement step, the extracted frames become easier to read when read on a mobile terminal such as a smartphone.

[0047] The rearrangement step in this embodiment will be described with reference to FIGS. Here, an example will be described in which the comic image 10 is read from the top left to the bottom right. In the example shown in FIG. 6, the images are read in the order of A, B, C, D, and E. The origin is (0,0), the horizontal direction is the x-axis, and the vertical direction is the y-axis. The top left points of each frame are a to e, and their respective values ​​are expressed as x1 <x2<x3であり、y1<y2<y3である。

[0048] In the first extraction step and the second extraction step, rearrangement is not performed and no numbering (also called numbering) is performed. Numbering may be performed when the frames are extracted in the first extraction step and the second extraction step, but since the order in which the frames are extracted may not correspond to the order in which the frames are read, it is preferable to perform a rearrangement step.

[0049] Fig. 7(A) is an example of the case where a rearrangement step has been performed. The rearrangement process can be modified as appropriate, but when reading from the top left to the bottom right, first the frames are rearranged in ascending order of y-axis value. Next, if there are frames with the same y-axis value, the frames with the same y-axis value are rearranged in descending order of x-axis value.

[0050] By rearranging the frames in this way, the order shown in Fig. 7(B) can be obtained. Here, the rearrangement results in the display numbers 1 to 5 being assigned. In the rearrangement process, the display numbers may or may not be assigned as described above. The display numbers can be used as the display order when displaying the extracted frames on a mobile terminal such as a smartphone.

[0051] In the sorting process in this embodiment, a step of adjusting the horizontal width may be performed on the extracted frames. By performing the horizontal width adjustment, when displaying on a mobile terminal such as a smartphone, the extracted frames become easier to read. In FIG. 7(B), an example of adjusting the horizontal width of each frame based on the horizontal width of frame B is shown. For example, since the horizontal widths of frames A and E are smaller than that of frame B, the horizontal widths are widened to be the same as that of frame B. At this time, it is preferable that the ratio of the vertical to the horizontal (aspect ratio) of frames A and E remains constant and the horizontal width is set to the reference size. By doing so, the picture within the frame can be displayed without being distorted. Similarly, the horizontal widths of frames C and D are narrowed to the reference size.

[0052] In this embodiment, the horizontal width adjustment is not essential and may be performed as appropriate. If necessary, instead of adjusting the horizontal width, portions that are insufficient in the reference size may be painted black. For example, in frame E whose horizontal width is smaller than the reference horizontal width (the horizontal width of frame B), instead of increasing the horizontal width, the portion that is insufficient in the reference horizontal width may be painted black. An example of this case is shown in FIG. 7(C). By doing so, it is possible to prevent the small frame from being enlarged too much and becoming difficult to see instead. Note that other colors such as white painting may be used instead of black painting.

[0053] In this embodiment, it is not limited to a comic image that reads from the upper left to the lower right. A comic image that reads from the upper right to the lower left may also be used. An example of this case will be described with reference to FIG. 8. FIG. 8 is the same figure as FIG. 6, in which frame extraction is performed on the comic image 10, sorting is performed, and the horizontal width is adjusted.

[0054] Here, in FIG. 8, the upper right points in each frame are set as a to e, and the respective values are such that x1 < x2 < x3 and y1 < y2 < y3. The sorting process can be changed as appropriate. However, when reading from the upper right to the lower left, first, the frames are sorted in the order of the frames with smaller y-axis values. Next, when there are frames with the same y-axis value, the frames are sorted in the order of the frames with larger x-axis values among the frames with the same y-axis value. By rearranging the images in this way, the comic images can be arranged in an order that follows the reading direction, similar to that of FIG. 6(C). In this manner, in this embodiment, even if a comic image is read from the upper right to the lower left, frames can be extracted and rearranged.

[0055] In the sorting process, frames that are too small may be removed as noise. For example, a "frame that is too small" may be a rectangular area drawn as part of a picture, and this rectangular area may be extracted.

[0056] The size of the frames to be removed as noise can be changed as appropriate; for example, frames whose width is 1 / 5 or less of the width of the comic image may be removed. In the case of normalization, frames whose width is 1 / 5 or less of the width of the comic image after normalization may be removed. By removing unnecessary images as noise, the comic image becomes easier to read. [Explanation of symbols]

[0057] 10 Cartoon Images 20 Border 30 Clustering Areas

Claims

1. 1. A computer-implemented method comprising: a first extraction step of extracting a contour of the comic image and extracting an area surrounded by a frame line in the comic image as a frame; a rearrangement step of rearranging the frames based on the positions of the frames in the comic image; having A new comic image is presented on a mobile terminal in which the rearranged frames are arranged vertically, and the widths of the frames in the new comic image are adjusted so as to approach the width of a reference frame, and the color tone is changed in areas where the width of the frames falls short of the reference size.

2. 10. The method of claim 1 , Other of the frames have their aspect ratios kept constant and their widths adjusted.

3. A program for causing a computer to execute each step of the method according to claim 1 or 2.

4. A computer comprising a processor, the computer being configured to execute each step of the method of claim 1 or 2 by the processor reading a program.

Images