Comic image processing method and device, storage medium, and electronic device

Abstract

The present disclosure provides a comic image processing method and device, electronic device, and storage medium, and relates to the field of image processing technology. The comic image processing method includes: extracting the contour of the comic image to obtain the contour information of the comic image; performing polygon fitting based on the contour information, and determining the storyboard images based on the polygons obtained by fitting; sorting the storyboard images according to the position information of each storyboard image in the comic image; generating storyboard information based on the position information and sorting information of each storyboard image, and associating the storyboard information with the comic image. The present disclosure can improve the accuracy of storyboarding of comic images.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to a comic image processing method, a comic image processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] With the popularity of various mobile terminals such as smart phones, tablet computers, and e-book readers, mobile reading has developed rapidly. Mobile reading solves the shortcomings of traditional reading, such as inconvenience in carrying and poor interactivity, and also provides the possibility of solving various reading barriers.

[0003] Comics are a relatively special type of publication, and their pages are usually divided into several independent sub-images. In the field of comic production, these independent sub-images are often called "storyboards." In order to facilitate users to read comics on mobile terminals, it is usually necessary to divide each comic into smaller pictures according to each storyboard, and display each storyboard in sequence according to the storyline.

[0004] In the related art, comics with obvious frame boundaries can be segmented based on lines. However, the frame boundaries of some comic works are not completely straight lines, and some frame outlines are polygons with more than 4 sides, resulting in unsatisfactory segmentation effects.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The embodiments of the present invention provide a comic image processing method, a comic image processing device, an electronic device, and a computer-readable storage medium, thereby overcoming the problem of unsatisfactory comic storyboard boundary segmentation effect caused by the limitations and defects of related technologies to a certain extent.

[0007] An embodiment of the present invention provides a comic image processing method, comprising:

[0008] Extracting contours of the comic image to obtain contour information of the comic image;

[0009] Performing polygon fitting based on the contour information, and determining the storyboard images based on the polygons obtained by fitting;

[0010] sorting the storyboard images according to position information of the storyboard images in the comic image;

[0011] Storyboard information is generated based on the position information and the sorting information of each of the storyboard images, and the storyboard information is associated with the comic image.

[0012] In an exemplary embodiment of the present disclosure, extracting contours from a comic image includes:

[0013] Binarizing the comic image to obtain a binary image;

[0014] Performing edge detection on the binary image to obtain an edge image;

[0015] The contour of the edge image is extracted to obtain contour information of the comic image.

[0016] In an exemplary embodiment of the present disclosure, performing polygon fitting based on the contour information includes:

[0017] Describing the outline of the comic image using continuous lines based on the outline information to generate a plurality of closed areas;

[0018] The closed areas in which there is an overlapping relationship are merged, and the closed areas in which there is a containing relationship are merged to obtain polygons.

[0019] In an exemplary embodiment of the present disclosure, determining the storyboard images based on the polygons obtained by fitting includes:

[0020] Determine the rectangular area where each polygon is located according to the maximum coordinate point and the minimum coordinate point of each polygon on the horizontal axis of the rectangular coordinate system, and the maximum coordinate point and the minimum coordinate point of each polygon on the vertical axis;

[0021] In the comic image, each storyboard image is extracted corresponding to the rectangular area where each polygon is located.

[0022] In an exemplary embodiment of the present disclosure, the method further includes:

[0023] When the number of storyboard images determined based on each polygon is not greater than 1, performing text recognition on the comic image to determine a text area in the comic image;

[0024] The comic image is cut into a plurality of storyboard images based on a segmentation line that does not intersect with each of the text areas.

[0025] In an exemplary embodiment of the present disclosure, the position information of the storyboard image in the comic image includes: the position coordinates of the storyboard image in the comic image and the width and height of the storyboard image.

[0026] In an exemplary embodiment of the present disclosure, the position coordinates include vertex coordinates, and the sorting of the storyboard images includes:

[0027] Obtaining a reading direction of the comic image;

[0028] The storyboard images are sorted according to the reading direction and the position of the vertex coordinates of each storyboard image in the rectangular coordinate system.

[0029] In an exemplary embodiment of the present disclosure, sorting the storyboard images according to the reading direction and the position of the vertex coordinates of the storyboard images in the rectangular coordinate system includes:

[0030] When the reading direction is from left to right, sorting the storyboard images in a sequence from left to right and from top to bottom based on the position of the upper left vertex of each storyboard image in the rectangular coordinate system;

[0031] When the reading direction is from right to left, the storyboard images are sorted in order from right to left and from top to bottom based on the position of the upper right vertex of each storyboard image in the rectangular coordinate system.

[0032] In an exemplary embodiment of the present disclosure, the storyboard information is generated based on the position coordinates of each of the storyboard images, the width and height of each of the storyboard images, and the sorting information of each of the storyboard images.

[0033] In an exemplary embodiment of the present disclosure, associating the storyboard information with the comic image includes:

[0034] The storyboard information is written into the file extension information area of ​​the comic image.

[0035] In an exemplary embodiment of the present disclosure, the comic image processing method further includes:

[0036] The current storyboard image to be displayed is used as the target storyboard image, and the target storyboard image is displayed separately in the designated display area.

[0037] In an exemplary embodiment of the present disclosure, the target storyboard image is displayed separately in a designated display area, including:

[0038] Based on the storyboard information of the target storyboard image, determining the area of ​​the target storyboard image in the comic image;

[0039] Scaling the comic image according to the size of the designated display area and the size of the target storyboard image;

[0040] Moving the area where the target storyboard image is located in the scaled comic image to the designated display area;

[0041] A mask layer is generated to cover the area outside the area where the target storyboard image is located.

[0042] The embodiment of the present invention further provides a comic image processing device, comprising:

[0043] A contour extraction module, used to extract the contour of the comic image and obtain the contour information of the comic image;

[0044] A storyboard determination module, used for performing polygon fitting based on the outline information, and determining storyboard images based on each polygon obtained by fitting;

[0045] A storyboard sorting module, used for sorting each of the storyboard images according to position information of each of the storyboard images in the comic image;

[0046] An information association module is used to generate storyboard information based on the position information and sorting information of each storyboard image, and associate the storyboard information with the comic image.

[0047] In an exemplary embodiment of the present disclosure, the contour extraction module includes:

[0048] A binarization unit, used for performing binarization processing on the comic image to obtain a binarized image;

[0049] An edge detection unit, used to perform edge detection on the binary image to obtain an edge image;

[0050] The contour extraction unit is used to extract the contour of the edge image to obtain contour information of the comic image.

[0051] In an exemplary embodiment of the present disclosure, the mirror determination module includes:

[0052] an outline drawing unit, configured to draw the outline of the comic image using continuous lines based on the outline information to generate a plurality of closed areas;

[0053] The region merging unit is used to merge closed regions that have overlapping relationships and to merge closed regions that have inclusion relationships to obtain polygons.

[0054] In an exemplary embodiment of the present disclosure, the mirror determination module further includes:

[0055] A rectangular area determination unit, used to determine the rectangular area where each polygon is located according to the maximum coordinate point and the minimum coordinate point of each polygon on the horizontal axis of the rectangular coordinate system, and the maximum coordinate point and the minimum coordinate point on the vertical axis;

[0056] The storyboard image extraction unit is used to extract each storyboard image in the comic image corresponding to the rectangular area where each polygon is located.

[0057] In an exemplary embodiment of the present disclosure, the comic image processing apparatus further includes:

[0058] A text recognition module, used for performing text recognition on the comic image when the number of storyboard images determined based on each polygon is not greater than 1, and determining a text area in the comic image;

[0059] The image segmentation module is used to cut the comic image into a plurality of storyboard images based on a segmentation line that does not intersect with each of the text areas.

[0060] In an exemplary embodiment of the present disclosure, the position information of the storyboard image in the comic image includes: the position coordinates of the storyboard image in the comic image and the width and height of the storyboard image.

[0061] In an exemplary embodiment of the present disclosure, the position coordinates include vertex coordinates, and the storyboard sorting module includes:

[0062] A reading direction obtaining unit, used to obtain the reading direction of the comic image;

[0063] The storyboard sorting unit is used to sort the storyboard images according to the reading direction and the position of the vertex coordinates of each storyboard image in the rectangular coordinate system.

[0064] In an exemplary embodiment of the present disclosure, the storyboard sorting unit is used to sort each of the storyboard images in an order from left to right and from top to bottom based on the position of the upper left vertex of each of the storyboard images in the rectangular coordinate system when the reading direction is from left to right; or is used to sort each of the storyboard images in an order from right to left and from top to bottom based on the position of the upper right vertex of each of the storyboard images in the rectangular coordinate system when the reading direction is from right to left.

[0065] In an exemplary embodiment of the present disclosure, the information association module includes:

[0066] The storyboard information generating unit is used to generate the storyboard information based on the position coordinates of each storyboard image, the width and height of each storyboard image, and the sorting information of each storyboard image.

[0067] In an exemplary embodiment of the present disclosure, the information association module includes:

[0068] The information writing unit is used to write the storyboard information into the file extension information area of ​​the comic image.

[0069] In an exemplary embodiment of the present disclosure, the comic image processing apparatus further includes:

[0070] The storyboard image display module is used to use the storyboard image to be displayed currently as the target storyboard image, and display the target storyboard image separately in a designated display area.

[0071] In an exemplary embodiment of the present disclosure, the storyboard image display module includes:

[0072] A comic area determination unit, configured to determine an area of ​​the target storyboard image in the comic image based on the storyboard information of the target storyboard image;

[0073] A comic image scaling unit, configured to scale the comic image according to the size of the designated display area and the size of the target storyboard image;

[0074] A comic image moving unit, used to move the area where the target storyboard image is located in the scaled comic image to the designated display area;

[0075] The mask layer is used to cover the area outside the area where the target storyboard image is located.

[0076] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned methods is implemented.

[0077] An embodiment of the present invention further provides an electronic device, including:

[0078] Processor; and

[0079] A memory, configured to store executable instructions of the processor;

[0080] The processor is configured to perform any of the above methods by executing the executable instructions.

[0081] On the one hand, this solution can accurately extract irregular-shaped storyboards by extracting the outline of the storyboard image and fitting polygons, thus ensuring the integrity of the storyboard image. On the other hand, this solution associates the location information and sorting information of the storyboard image with the comic image, and displays each storyboard image in a complete comic image in sequence based on the associated information, without actually cutting the comic image to generate each storyboard image file, thus avoiding the problem of redundant image files occupying memory and affecting performance, thereby reducing the consumption of storage resources while optimizing the user experience.

[0082] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0084] Figure 1 A schematic diagram showing an exemplary system architecture of a comic image processing method and device to which the embodiments of the present disclosure can be applied;

[0085] Figure 2 A schematic diagram showing the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown;

[0086] Figure 3 A flowchart of a comic image processing method according to an embodiment of the present disclosure is schematically shown;

[0087] Figure 4-Figure 8 A schematic diagram showing the effect of various steps of processing a comic image according to a comic image processing method in one embodiment of the present disclosure;

[0088] Fig. 9 A flowchart schematically shows the detailed steps of storyboarding a comic image according to a comic image processing method in one embodiment of the present disclosure;

[0089] Fig.10 A schematic diagram of an operation method for opening a storyboard browsing mode by double-clicking according to an embodiment of the present disclosure is shown;

[0090] Fig.11 A schematic diagram schematically shows a storyboard image displayed in a storyboard browsing mode according to an embodiment of the present disclosure;

[0091] Fig.12 A schematic diagram of an operation method for opening / exiting a storyboard browsing mode by setting an option button according to an embodiment of the present disclosure is shown;

[0092] Figure 13-Figure 15 A schematic diagram of an operation method for switching between two front and rear storyboard images in a storyboard browsing mode according to an embodiment of the present disclosure is schematically shown;

[0093] Fig.16 The block diagram schematically shows a comic image processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0094] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0095] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0096] Figure 1 A schematic diagram of a system architecture of an exemplary application environment in which a comic image processing method and apparatus according to an embodiment of the present disclosure can be applied is shown.

[0097] like Figure 1 As shown, the system architecture 100 may include one or more of terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc. The terminal devices 101, 102, 103 may be various electronic devices with display screens, including but not limited to desktop computers, portable computers, smart phones, tablet computers, etc. It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0098] The comic image processing method provided in the embodiment of the present disclosure is generally executed by the server 105, and accordingly, the comic image processing device is generally set in the server 105. However, it is easy for those skilled in the art to understand that the comic image processing method provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, and 103, and accordingly, the comic image processing device can also be set in the terminal devices 101, 102, and 103, and this is not particularly limited in the present exemplary embodiment. For example, in an exemplary embodiment, the user can upload the image to be fused to the server 105 through the terminal devices 101, 102, and 103, and the server fuses the face in the image to be fused into the template image through the comic image processing method provided in the embodiment of the present disclosure to obtain the target image, and transmits the target image to the terminal devices 101, 102, 103, etc.

[0099] Figure 2 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present disclosure is shown.

[0100] It should be noted that Figure 2 The computer system 200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0101] like Figure 2 As shown, the computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 202 or a program loaded from a storage part 208 into a random access memory (RAM) 203. Various programs and data required for system operation are also stored in the RAM 203. The CPU 201, the ROM 202, and the RAM 203 are connected to each other via a bus 204. An input / output (I / O) interface 205 is also connected to the bus 204.

[0102] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, etc.; an output section 207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 208 including a hard disk, etc.; and a communication section 209 including a network interface card such as a LAN card, a modem, etc. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 210 as needed, so that a computer program read therefrom is installed into the storage section 208 as needed.

[0103] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 209, and / or installed from the removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the various functions defined in the method and apparatus of the present application are executed. In some embodiments, the computer system 200 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0104] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0105] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0106] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0107] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being installed in the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiment. For example, the electronic device may implement the following method. Figure 3 to Figure 15 The various steps shown, etc.

[0108] The technical solution of the embodiment of the present disclosure is described in detail below:

[0109] In the related technology, a storyboard recognition method based on line segmentation is used to realize the storyboard of comic images, and a storyboard sorting method based on a binary tree is used. The storyboard recognition of this solution is only applicable to comics with obvious storyboard boundaries and storyboards in the form of quadrilaterals. However, in reality, the storyboard boundaries of some comic works are not completely straight lines, and some storyboard outlines are polygons with more than four sides, which makes the application scenarios of this solution relatively limited.

[0110] This exemplary embodiment provides a comic image processing method. The comic image processing method can be applied to the server 105, or to one or more of the terminal devices 101, 102, 103, which is not particularly limited in this exemplary embodiment. Figure 3As shown, the comic image processing method may include the following steps S310 to S340:

[0111] Step S310. Extract the contour of the comic image to obtain contour information of the comic image;

[0112] Step S320: performing polygon fitting based on the contour information, and determining the storyboard images based on the polygons obtained by fitting;

[0113] Step S330. Sort the storyboard images according to the position information of the storyboard images in the comic image;

[0114] Step S340: Generate storyboard information based on the position information and sorting information of each storyboard image, and associate the storyboard information with the comic image.

[0115] The comic image processing method provided in this example implementation can, on the one hand, accurately extract irregular-shaped storyboards by extracting the outline of the storyboard image and fitting polygons, thereby ensuring the integrity of the storyboard image. On the other hand, the position information and sorting information of the storyboard image are associated with the comic image, and the storyboard images in a complete comic image are displayed in sequence according to the associated information, without actually cutting the comic image to generate each storyboard image file, thereby avoiding the problem of redundant image files occupying memory and affecting performance, thereby reducing the consumption of storage resources while optimizing the user experience.

[0116] Figure 4-Figure 8 The effect diagram schematically shows the various steps of processing a comic image according to a comic image processing method in one embodiment of the present disclosure.

[0117] Below, reference Figure 4-Figure 8 , the above steps of this example implementation are explained in more detail.

[0118] In step S310, contour extraction is performed on the comic image to obtain contour information of the comic image.

[0119] In this example implementation, the comic image may have one or more storyboard images. The outline of each storyboard image may be a polygon, and the number of sides of each polygon may be three, four, five or even more. In addition, in some comic images, the boundaries of some storyboard images are not completely straight lines, and the boundaries between the storyboard images are not obvious. The comic images involved in this example implementation include but are not limited to the comic images described above, and the present disclosure is not limited thereto.

[0120] In this example implementation, the comic image to be processed can be Figure 4As shown, the figure includes five storyboard images, and the shape of each storyboard image includes a regular or irregular trapezoid.

[0121] In step S311, the comic image can first be binarized to obtain a binary image. In digital image processing, the binarization of the image can remove color information and highlight the target outline, which is beneficial to the further processing of the image, making the image simpler and reducing the amount of data. Specifically, the grayscale of pixels greater than a certain critical grayscale value can be set as the grayscale maximum value, and the grayscale of pixels less than this value can be set as the grayscale minimum value, so that the entire image presents an obvious visual effect of only black and white. The binary image obtained by the binarization process in this step can be as follows Figure 5 shown.

[0122] In step S312, edge detection may be performed on the binary image to obtain an edge image. Further, the target edge in the binary image may be detected to identify the outline in the comic picture. Specifically, an edge detection algorithm may be used to identify the target edge. The edge image obtained by edge detection in this step may be as follows: Figure 6 shown.

[0123] In step S313, the contour of the edge image may be extracted to obtain contour information of the comic image. Figure 6 The edge image in the image is used for contour extraction, and edge points can be found in sequence to track the boundary. It is easy for those skilled in the art to understand that in other exemplary embodiments of the present disclosure, other methods can also be used for contour extraction to obtain contour information of the comic image.

[0124] In step S320, polygon fitting is performed based on the contour information, and the storyboard images are determined based on the polygons obtained by fitting. Specifically, the following steps may be included:

[0125] In step S321, the outline of the comic image is drawn using continuous lines based on the outline information to generate a plurality of closed areas.

[0126] In this exemplary embodiment, continuous lines can be used to Figure 6 The outline of the comic image in is drawn to generate multiple closed areas. Figure 6 In the process, characters, text boxes, background patterns and even the outline of the entire storyboard image can be circled out through line drawing.

[0127] In step S322, the closed areas in which there is an overlapping relationship are merged, and the closed areas in which there is a containment relationship are merged to obtain each polygon. Furthermore, among the multiple closed areas obtained in step S321, there may be areas that overlap or contain each other. For example, there may be multiple objects such as characters, objects, and text boxes in a comic image, and two or more objects may have some common areas, which is an overlapping relationship. Even if the area of ​​one object is entirely located in the area of ​​another object, it is a containment area. The above-mentioned overlapping or contained areas can be merged. That is, the areas of two objects in an overlapping relationship are merged, and the area obtained after the merger includes the entire areas of the two objects; the areas of two objects in a containment relationship are merged, and the area obtained after the merger is the area of ​​the object with the larger area. In this way, after merging multiple areas in an overlapping relationship or a containment relationship, it is possible to eventually generate an area corresponding to a storyboard image, and further obtain the polygon. As Figure 7 As shown, it is the effect diagram after polygon fitting, and each polygon finally generated is located in each corresponding mirror image area.

[0128] In step S323, the rectangular area where each polygon is located is determined based on the maximum coordinate point and the minimum coordinate point of each polygon on the horizontal axis of the rectangular coordinate system, and the maximum coordinate point and the minimum coordinate point of each polygon on the vertical axis. The polygons obtained in step S322 include some irregular polygons, such as Figure 7 In this step, the corresponding complete storyboard image can be extracted for each polygon. At the same time, in order to facilitate the subsequent sorting of each storyboard image and the display of each storyboard image in the display area of ​​the terminal display screen, the rectangular area where each polygon is located can be determined based on the maximum coordinate point and the minimum coordinate point of each polygon on the horizontal axis of the rectangular coordinate system and the maximum coordinate point and the minimum coordinate point of each polygon on the vertical axis. The rectangular area where each polygon is located can be referred to Figure 8 As shown in Figure 8 The rectangular frames shown in .

[0129] In step S324, in the comic image, corresponding to the rectangular area where each polygon is located, each storyboard image is extracted. Further, each storyboard image can be extracted in the comic image according to the rectangular area where each polygon is located determined in step S323. Figure 8 As shown in , the storyboard images corresponding to each rectangular area may be 810-850 respectively.

[0130] In some embodiments, after step S320, if the number of storyboard images obtained is not greater than 1, it means that the storyboard images in the comic image cannot be identified. In order to keep the reading mode in the storyboard mode and ensure the browsing quality of the user, the comic image needs to be storyboarded while retaining the integrity of each text. The following steps may correspond to Fig. 9 Step S971 and step S972 of the flowchart:

[0131] In step S971, when the number of storyboard images determined based on each polygon is not greater than 1, text recognition can be performed on the comic image to determine the text area in the comic image. In an exemplary embodiment of the present disclosure, text recognition technology can be used to recognize text in an image, and then the area where the text is located can be determined. The text recognition technology can use OCR (Optical Character Recognition) to convert text in an image into a text format. Text recognition can also be performed on a comic image in other ways, and this exemplary embodiment is not limited to this.

[0132] In this example embodiment, after determining each storyboard image, it is also necessary to determine the reading direction of each storyboard image in the same comic image, so that when the storyboard images are switched in the storyboard browsing mode, they are switched according to the correct reading direction.

[0133] In step S330, the storyboard images are sorted according to the position information of each storyboard image in the comic image.

[0134] Further, the position information in the comic image may include the position coordinates of the storyboard image in the comic image and the width and height of the storyboard image. Here, the storyboard image may refer to the storyboard image determined in step S324 according to the rectangular area where each polygon is located, such as Figure 8 Specifically, the step of sorting the storyboard images may include:

[0135] In step S331, the reading direction of the comic image is obtained. In this exemplary embodiment, according to the reading direction of each adjacent comic image, the reading direction of the comic image can be from left to right or from right to left.

[0136] In step S332, the storyboard images are sorted according to the reading direction and the position of the vertex coordinates of each storyboard image in the rectangular coordinate system.

[0137] In this example embodiment, the position coordinates may include the vertex coordinates of each storyboard image, that is, the position of the vertex of each storyboard image in a rectangular coordinate system, and may specifically refer to the vertex coordinates of the rectangle corresponding to the storyboard image. Further, in this example embodiment, the vertex coordinates may also be divided into upper left vertex coordinates and upper right vertex coordinates. In other embodiments, the position coordinates may also include the center point coordinates of the area where each storyboard image is located, or the center of gravity coordinates, etc. Any feature points that can be used to distinguish storyboard images at different positions may be used as position coordinates, and the present disclosure does not make any special limitations hereon.

[0138] Specifically, when the reading direction of the comic image is from left to right, the storyboard images can be sorted from left to right and from top to bottom based on the position of the upper left vertex of the rectangle corresponding to each storyboard image in the rectangular coordinate system. Figure 4 The reading direction of the comic image shown in is from left to right, then the order can be based on the upper left vertex of each storyboard image, in the order from left to right and from top to bottom. The order result is referenced Figure 8 As shown, there are storyboard image 810, storyboard image 820, storyboard image 830, storyboard image 840 and storyboard image 850 in sequence.

[0139] In addition, when the reading direction of the comic image is from right to left, the storyboard images can be sorted from right to left and from top to bottom based on the position of the upper right vertex of the rectangle corresponding to each storyboard image in the rectangular coordinate system.

[0140] In step S340, storyboard information is generated based on the position information and the sorting information of each of the storyboard images, and the storyboard information is associated with the comic image.

[0141] In this example embodiment, the position information in the comic image may include the position coordinates of the storyboard image in the comic image and the width and height of the storyboard image. The position coordinates of the storyboard image may refer to the position coordinates as described in step S332. The width and height of the storyboard image may refer to the width and height of the storyboard image determined according to the rectangular area where each polygon is located in step S324, that is, Figure 8 The ordering information of the storyboard images may refer to the ordering results of the storyboard images obtained in step S332. In this example embodiment, the storyboard information may be generated based on the position coordinates of the storyboard images, the width and height of the storyboard images, and the ordering results of the storyboard images.

[0142] Furthermore, this exemplary embodiment can also associate the storyboard information with the comic image. Usually, comic images are mostly in picture format, and common picture formats such as JPG, PNG, and GIF are designed with an extended information area for storing extended information fields. In this exemplary embodiment, the above-mentioned storyboard information can be written into the file extended information area of ​​the comic image, and each storyboard image in a complete comic image can be displayed in sequence according to the storyboard information.

[0143] On the one hand, there is no need to actually cut the comic image to generate each storyboard image file, which avoids the problem of redundant image files occupying memory and affecting performance, thereby reducing the consumption of storage resources while optimizing the user experience. On the other hand, since the storyboard information is directly written in the corresponding comic image file, it can effectively avoid the problem of abnormal storyboard display caused by the comic image being updated but the storyboard information not being updated synchronously.

[0144] In the above embodiment, the storyboard image is determined for the comic image and the storyboard information is associated with the comic image. Fig. 9 A flowchart showing detailed steps of storyboarding a comic image according to a comic image processing method in one embodiment of the present disclosure is shown. The specific process may include:

[0145] Step S910, comic image processing starts; Step S921, image binarization; Step S922, obtaining contour information; Step S931, polygon fitting of the contour information; Step S932, determining the storyboard image; Step S940, determining whether the number of storyboard images is greater than 1; Step S950, when the number of storyboard images is greater than 1, sorting the reading of each storyboard image according to the reading direction; Step S971, when the number of storyboard images is not greater than 1, determining the text area using text recognition technology; Step S972, avoiding the text area and cutting the comic image; Step S960, outputting relevant storyboard information and associating the storyboard information with the comic image.

[0146] After the storyboard information is associated with the comic image, the following steps may be performed when displaying the storyboard image, including:

[0147] In step S350, the current storyboard image to be displayed is used as the target storyboard image, and the target storyboard image is displayed separately in the designated display area. The designated display area can be set to an area on the display screen that is within a preset area range and has a preset shape. For example, the designated display area can be a rectangular area located in the middle of the display screen, and the width and height are both within the width and height range of the display screen. In addition, the designated display area can also be a floating window that can be moved within the screen range and can be scalable, etc., which is not specifically limited in the present disclosure.

[0148] In step S351, in this example embodiment, the area of ​​the target storyboard image in the comic image can be determined based on the storyboard information of the target storyboard image. Specifically, the position of the target storyboard image in the comic image can be first determined based on the position coordinates of the target storyboard image in the storyboard information; then, based on the width and height of the storyboard image in the storyboard information, the position coordinates are used as reference points to extend the corresponding width horizontally and the corresponding height vertically to determine the area of ​​the target storyboard image in the comic image. More specifically, when the coordinate position is the upper left vertex coordinate, the corresponding width can be extended horizontally to the right and the corresponding height can be extended vertically downward, or when the coordinate position is the upper right vertex coordinate, the corresponding width can be extended horizontally to the left and the corresponding height can be extended vertically downward.

[0149] In step S352, the comic image is scaled according to the size of the designated display area and the size of the target storyboard image. After determining the area of ​​the target storyboard image in the comic image, the width and height of the area are known, and the size of the designated display area is compared. The comic image can be scaled according to the size of the designated display area and the size of the target storyboard image so that the comic image fits the size of the designated display area. In this example embodiment, when the designated display area is a rectangular area located in the middle of the display screen and the width and height are both within the width and height range of the display screen, the target storyboard image can be scaled so that the width and height of the target storyboard image are not greater than the width and height of the rectangular area.

[0150] More specifically, when the ratio of the width to the height of the target storyboard image is not less than the ratio of the width to the height of the designated display area, the target storyboard image can be scaled to a width that is consistent with the designated display area based on the width of the designated display area, and the height of the target storyboard image is not greater than the height of the designated display area, so that the storyboard image is displayed in the designated display area at the maximum size, and the integrity of the entire storyboard image is ensured. When the ratio of the width to the height of the target storyboard image is not greater than the ratio of the width to the height of the designated display area, the target storyboard image can be scaled to a height that is consistent with the designated display area based on the height of the designated display area, and the width of the target storyboard image is not greater than the width of the designated display area, so that the storyboard image is displayed in the designated display area at the maximum size, and the integrity of the entire storyboard image is ensured.

[0151] In step S353, the area where the target storyboard image in the scaled comic image is located is moved to the designated display area. The scaled target storyboard image obtained in step S352 is moved to the designated display area according to the position information of the designated display area. The position information of the designated display area may include the position coordinates of the feature points of the display area and the width and height of the area. For example, when the display area is a rectangle, the coordinates of each vertex of the rectangular display area, or the coordinates of one vertex and the width and height thereof, may be determined to determine the position and range of the rectangular display area, and then the target storyboard image is moved into the area.

[0152] In step S354, a mask layer is generated to cover the area outside the area where the target storyboard image is located. In order to enhance the user's immersive reading experience and present a full-screen immersive browsing effect, a mask layer can be set to cover the area outside the target storyboard image. The mask layer can be scaled to adapt to the screen width and height and cover the area outside the target storyboard image based on the current location and size information of the target storyboard image obtained in the aforementioned steps. Typically, the mask layer can be black, dark gray, or other colors, which are not specifically limited in the present disclosure. The display effect of a storyboard image that is ultimately in storyboard browsing mode can be as follows: Fig.11 shown.

[0153] In this example embodiment, a common image browsing tool such as Matrix can be used to move the storyboard image to the center of the screen, and the width and height can be scaled to fit the screen width and height. Finally, a mask area overlaid on the reader can be set to block the content outside the current storyboard area.

[0154] In addition, the comic image processing method provided by the present disclosure may also include methods for opening and exiting the browsing mode, and switching between adjacent storyboard images in the browsing mode. Figure 10-15 , which may include:

[0155] 1) Opening and exiting browsing mode

[0156] When the user is browsing comics, double-clicking any area of ​​the comic image will enter the storyboard browsing mode. Fig.10 , Fig.11 As shown in the figure, double-click to enter the browsing mode, and the first storyboard image displayed is the storyboard image where the double-click gesture is located. In the storyboard browsing mode, double-click any area of ​​the screen to exit and zoom back to the full comic page where the storyboard is located.

[0157] In addition, you can also enable or exit the storyboard browsing mode by setting the option button. Fig.12 As shown, the option button can be set at one end of the comic reading screen. The button is set with two options: on and off. The storyboard browsing mode can be turned on or off by clicking or dragging the button.

[0158] 2) Browse comics in storyboard mode

[0159] When browsing comics in the storyboard mode, you can switch between adjacent storyboard images through preset gestures. The preset gestures may include sliding in various directions. Figure 13-15 As shown, Fig.13 The current storyboard image is being browsed. When switching from the current storyboard image to the next storyboard image, the switching gesture can be as follows: Fig.14 Swipe left, then Fig.15 The next storyboard image browsed after the switch. Conversely, when switching from the current storyboard image to the previous storyboard image, the switching gesture may be a rightward slide. In addition, in other embodiments, the switching gesture may also be an upward or downward slide, which is not specifically limited in the present disclosure.

[0160] 3) Other functions

[0161] When browsing comic images, you can also adjust the screen display brightness through settings, or adjust the mask layer color, brightness, etc. through settings.

[0162] Fig.16 The block diagram schematically shows a comic image processing apparatus 1600 according to an embodiment of the present disclosure.

[0163] Furthermore, in this exemplary embodiment, a comic image processing apparatus 1600 is also provided. The comic image processing apparatus 1600 can be applied to a server or a terminal device. Fig.16 As shown, the comic image processing device 1600 may include a contour extraction module 1610, a storyboard determination module 1620, a storyboard sorting module 1630, and an information association module 1640. Among them:

[0164] The contour extraction module 1610 is used to extract the contour of the comic image and obtain the contour information of the comic image;

[0165] A storyboard determination module 1620, configured to perform polygon fitting based on the contour information, and determine storyboard images based on the polygons obtained by fitting;

[0166] A storyboard sorting module 1630, configured to sort the storyboard images according to position information of the storyboard images in the comic image;

[0167] The information association module 1640 is used to generate storyboard information based on the position information and sorting information of each storyboard image, and associate the storyboard information with the comic image.

[0168] In an exemplary embodiment of the present disclosure, the contour extraction module 1610 includes:

[0169] A binarization unit, used for performing binarization processing on the comic image to obtain a binarized image;

[0170] An edge detection unit, used to perform edge detection on the binary image to obtain an edge image;

[0171] The contour extraction unit is used to extract the contour of the edge image to obtain contour information of the comic image.

[0172] In an exemplary embodiment of the present disclosure, the mirror determination module 1620 includes:

[0173] an outline drawing unit, configured to draw the outline of the comic image using continuous lines based on the outline information to generate a plurality of closed areas;

[0174] The area merging unit is used to merge the closed areas that overlap or contain each other to obtain polygons.

[0175] In an exemplary embodiment of the present disclosure, the mirror determination module 1620 further includes:

[0176] A rectangular area determination unit, used to determine the rectangular area where each polygon is located according to the maximum coordinate point and the minimum coordinate point of each polygon on the horizontal axis and the maximum coordinate point and the minimum coordinate point on the vertical axis of the rectangular coordinate system;

[0177] The storyboard image extraction unit is used to extract each storyboard image in the comic image corresponding to the rectangular area where each polygon is located.

[0178] In an exemplary embodiment of the present disclosure, the comic image processing device 1600 further includes:

[0179] A text recognition module, used for performing text recognition on the comic image when the number of storyboard images determined based on each polygon is not greater than 1, and determining a text area in the comic image;

[0180] The image segmentation module is used to cut the comic image into a plurality of storyboard images based on a plurality of segmentation lines that do not intersect with each of the text areas.

[0181] In an exemplary embodiment of the present disclosure, the position information of the storyboard image in the comic image includes: the position coordinates of the storyboard image in the comic image and the width and height of the storyboard image.

[0182] In an exemplary embodiment of the present disclosure, the position coordinates include vertex coordinates, and the storyboard sorting module 1630 includes:

[0183] A reading direction obtaining unit, used to obtain the reading direction of the comic image;

[0184] The storyboard sorting unit is used to sort the storyboard images according to the reading direction and the position of the vertex coordinates of each storyboard image in the rectangular coordinate system.

[0185] In an exemplary embodiment of the present disclosure, the storyboard sorting unit is used to sort each of the storyboard images in an order from left to right and from top to bottom based on the position of the upper left vertex of each of the storyboard images in the rectangular coordinate system when the reading direction is from left to right; or is used to sort each of the storyboard images in an order from right to left and from top to bottom based on the position of the upper right vertex of each of the storyboard images in the rectangular coordinate system when the reading direction is from right to left.

[0186] In an exemplary embodiment of the present disclosure, the information association module 1640 includes: a storyboard information generation unit, used to generate the storyboard information based on the position coordinates of each storyboard image, the width and height of each storyboard image, and the sorting information of each storyboard image.

[0187] In an exemplary embodiment of the present disclosure, the information association module 1640 includes: an information writing unit, configured to write the storyboard information into an extended information area of ​​the comic image file.

[0188] In an exemplary embodiment of the present disclosure, the comic image processing device 1600 further includes: a storyboard image display module, configured to use the storyboard image to be displayed currently as a target storyboard image, and to display the target storyboard image separately in a designated display area.

[0189] In an exemplary embodiment of the present disclosure, the storyboard image display module includes:

[0190] A comic area determination unit, configured to determine an area of ​​the target storyboard image in the comic image based on the storyboard information of the target storyboard image;

[0191] A comic image scaling unit, configured to scale the comic image according to the size of the designated display area and the size of the target storyboard image;

[0192] A comic image moving unit, used to move the area where the target storyboard image is located in the scaled comic image to the designated display area;

[0193] The mask layer is used to cover the area outside the area where the target storyboard image is located.

[0194] The specific details of each module or unit in the comic image processing device 1600 have been described in detail in the corresponding comic image processing method, so they will not be repeated here.

[0195] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0196] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0197] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A comic image processing method, characterized in that: include: Extracting contours of the comic image to obtain contour information of the comic image; Performing polygon fitting based on the contour information, and determining storyboard images based on each polygon obtained by fitting, and when the number of storyboard images determined based on each polygon is not greater than 1, performing text recognition on the comic image to determine a text area in the comic image; and cutting the comic image into a plurality of storyboard images based on a segmentation line that does not intersect each of the text areas; sorting the storyboard images according to position information of the storyboard images in the comic image; The storyboard information is generated based on the position information and sorting information of each storyboard image, and the storyboard information is written into the file extension information area of ​​the comic image, and each storyboard image in the comic image is displayed in sequence according to the storyboard information.

2. The comic image processing method according to claim 1, characterized in that: The contour extraction of the comic image comprises: Binarizing the comic image to obtain a binary image; Performing edge detection on the binary image to obtain an edge image; The contour of the edge image is extracted to obtain contour information of the comic image.

3. The comic image processing method according to claim 1 or 2, characterized in that: Performing polygon fitting based on the contour information includes: Describing the outline of the comic image using continuous lines based on the outline information to generate a plurality of closed areas; The closed areas in which there is an overlapping relationship are merged, and the closed areas in which there is a containing relationship are merged to obtain polygons.

4. The comic image processing method according to claim 1, characterized in that: Determining the split-mirror images based on the polygons obtained by fitting comprises: Determine the rectangular area where each polygon is located according to the maximum coordinate point and the minimum coordinate point of each polygon on the horizontal axis of the rectangular coordinate system, and the maximum coordinate point and the minimum coordinate point of each polygon on the vertical axis; In the comic image, each storyboard image is extracted corresponding to the rectangular area where each polygon is located.

5. The comic image processing method according to claim 1, characterized in that: The position information of the storyboard image in the comic image includes: the position coordinates of the storyboard image in the comic image and the width and height of the storyboard image.

6. The comic image processing method according to claim 5, characterized in that: The position coordinates include vertex coordinates, and the sorting of the storyboard images includes: Obtaining a reading direction of the comic image; The storyboard images are sorted according to the reading direction and the position of the vertex coordinates of each storyboard image in a rectangular coordinate system.

7. The comic image processing method according to claim 6, characterized in that: Sorting the storyboard images according to the reading direction and the position of the vertex coordinates of the storyboard images in the rectangular coordinate system includes: When the reading direction is from left to right, sorting the storyboard images in a sequence from left to right and from top to bottom based on the position of the upper left vertex of each storyboard image in the rectangular coordinate system; When the reading direction is from right to left, the storyboard images are sorted in order from right to left and from top to bottom based on the position of the upper right vertex of each storyboard image in the rectangular coordinate system.

8. The comic image processing method according to claim 7, characterized in that: The storyboard information is generated based on the position coordinates of each of the storyboard images, the width and height of each of the storyboard images, and the sorting information of each of the storyboard images.

9. The comic image processing method according to claim 1, characterized in that: Also includes: The current storyboard image to be displayed is used as the target storyboard image, and the target storyboard image is displayed separately in the designated display area.

10. The comic image processing method according to claim 9, characterized in that: Displaying the target storyboard image separately in a designated display area includes: Based on the storyboard information of the target storyboard image, determining the area of ​​the target storyboard image in the comic image; Scaling the comic image according to the size of the designated display area and the size of the target storyboard image; Moving the area where the target storyboard image is located in the scaled comic image to the designated display area; A mask layer is generated to cover the area outside the area where the target storyboard image is located.

11. A comic image processing device, characterized in that: include: A contour extraction module, used to extract the contour of the comic image and obtain the contour information of the comic image; A storyboard determination module, used for performing polygon fitting based on the outline information, and determining storyboard images based on each polygon obtained by fitting; A storyboard sorting module, used for sorting each of the storyboard images according to position information of each of the storyboard images in the comic image; An information association module, configured to generate storyboard information based on the position information and the sorting information of each storyboard image, and associate the storyboard information with the comic image; The information association module includes: an information writing unit, which is used to write the storyboard information into the extended information area of ​​the comic image file, and sequentially display each storyboard image in the comic image according to the storyboard information; The comic image processing device further includes: A text recognition module, used for performing text recognition on the comic image when the number of storyboard images determined based on each polygon is not greater than 1, and determining a text area in the comic image; The image segmentation module is used to cut the comic image into a plurality of storyboard images based on a plurality of segmentation lines that do not intersect with each of the text areas.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 9 by executing the executable instructions.

14. A computer program product, characterized in that The method comprises a computer program carried on a computer-readable medium, wherein the computer program contains a program code for executing the method according to any one of claims 1 to 9.

Images