Image-text mixed typesetting method and device

Abstract

The present application provides a method and device for mixed typesetting of text and images, the method comprising: obtaining the original text and image resources to be processed, the original text and image resources including picture book publication resources; parsing the original text and image resources to obtain the text resources and image resources to be processed; determining the text arrangement features corresponding to the text resources according to the layout type to which the text resources belong; performing mixed typesetting processing on the text resources and the image resources according to the text arrangement features, and displaying the processed mixed typesetting of text and images resources in the interactive interface of the client, so as to provide interactive functions to users through the mixed typesetting of text and images resources in the interactive interface. The method determines the text arrangement features corresponding to the text resources in a targeted manner according to the layout type to which the text resources belong, so that the mixed typesetting of text resources and image resources can be more accurately realized according to the text arrangement features, thereby improving the rendering accuracy of the final mixed typesetting of text and images resources, improving the efficiency of mixed typesetting of text and images, and improving the user interaction experience.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method and device for mixing images and texts. Background Art

[0002] With the development of smart terminals, online teaching methods based on smart terminals are becoming popular. At present, it mainly uses smart software installed on terminal devices as a carrier to present various courses to be learned through terminal devices, so that users can learn courses anytime and anywhere.

[0003] Taking picture book reading software as an example, in order to enrich the display content, it is necessary to present pictures and texts simultaneously in the terminal device by mixing pictures and texts. In related technologies, H5 front-end rendering technology can be used to achieve mixed pictures and texts, and dynamically load and render picture book pages through web views (Web View). However, the rendering process of this mixed picture and text method is slow, and the efficiency of mixed picture and text is low.

[0004] In summary, it is urgent to propose a technical solution to overcome the above-mentioned technical problems existing in the related technologies. Summary of the invention

[0005] The present application provides a method and device for mixing graphics and text to improve the rendering accuracy of graphics and text resources, enhance the efficiency of graphics and text mixing, and improve the user interaction experience.

[0006] In a first aspect, the present application provides a method for mixed typesetting of images and texts, the method comprising:

[0007] Obtaining original graphic resources to be processed, the original graphic resources including picture book publication resources;

[0008] Parsing the original graphic resources to obtain text resources and image resources to be processed;

[0009] Determining text arrangement features corresponding to the text resource according to the layout type to which the text resource belongs;

[0010] The text resource and the image resource are mixed according to the text arrangement feature, and the processed mixed text and image resource is displayed in the interactive interface of the client, so as to provide interactive functions to the user through the mixed text and image resource in the interactive interface.

[0011] In a second aspect, an embodiment of the present application provides a device for mixing text and images, the device comprising:

[0012] An acquisition module, used to acquire original graphic resources to be processed, the original graphic resources including picture book publication resources;

[0013] A parsing module, used to parse the original graphic resources to obtain text resources and image resources to be processed;

[0014] A determination module, used to determine the text arrangement feature corresponding to the text resource according to the layout type to which the text resource belongs;

[0015] The mixed display module is used to mix the text resources and the image resources according to the text arrangement characteristics, and display the processed mixed text and image resources in the interactive interface of the client, so as to provide interactive functions to users through the mixed text and image resources in the interactive interface.

[0016] In a third aspect, the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the image and text mixing method described in any one of the first aspects above is implemented.

[0017] In a fourth aspect, a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute any of the image-text typesetting methods described in the first aspect.

[0018] In the technical solution provided in the embodiment of the present application, the original graphic resources to be processed are obtained, and the original graphic resources include picture book publication resources; the original graphic resources are parsed to obtain the text resources and image resources to be processed; the text arrangement features corresponding to the text resources are determined according to the layout type to which the text resources belong; the text resources and image resources are mixed according to the text arrangement features, and the processed graphic and text mixed resources are displayed in the interactive interface of the client, so as to provide interactive functions to users through the graphic and text mixed resources in the interactive interface. In this solution, the text arrangement features corresponding to the text resources are determined in a targeted manner according to the layout type to which the text resources belong, so that the mixed typesetting of text resources and image resources can be more accurately achieved according to the text arrangement features, thereby improving the rendering accuracy of the final graphic and text mixed resources, improving the efficiency of graphic and text mixed typesetting, and improving the user interaction experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a flowchart of a method for mixed typesetting of images and texts in an embodiment of the present application;

[0021] Figure 2 It is a schematic diagram of the principle of a method for mixed typesetting of images and texts in an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of an interactive interface of an embodiment of the present application;

[0023] Figure 4 is a schematic diagram of another interactive interface of an embodiment of the present application;

[0024] Figure 5 is a schematic diagram of a device for mixing text and images according to an embodiment of the present application;

[0025] Figure 6 It is a structural schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein in the specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

[0028] At present, online teaching methods mainly use intelligent software installed on terminal devices as the carrier, and various courses to be learned are presented through terminal devices, so that users can study courses anytime and anywhere.

[0029] Taking picture book reading software as an example, in order to enrich the display content, it is necessary to present pictures and texts simultaneously in the terminal device through mixed text and pictures. In the related technology, on the one hand, H5 front-end rendering technology can be used to achieve mixed text and pictures, and dynamically load and render picture book pages through Web View. However, the rendering process of this mixed text and pictures method is slow, and the efficiency of mixed text and pictures is low. In addition, due to the low rendering efficiency, it is easy to cause the phenomenon of voice and text being out of sync, affecting the user interaction experience.

[0030] On the other hand, in the picture book display page, text is drawn in a designated rectangular area in paragraphs. However, this mixed text and image layout cannot detect whether a word or phrase in a paragraph is hit, and it is difficult to accurately calculate the interaction position in the picture book reading scenario, which affects the user's interactive experience.

[0031] In order to solve at least one of the above technical problems, an embodiment of the present application provides a mixed text and image typesetting solution.

[0032] The text-and-graphics mixed typesetting solution provided in the embodiment of the present application can be executed by an electronic device, which can be a server, a server cluster, or a cloud server. The electronic device can be a terminal device such as a computer, a tablet computer, or a mobile phone. In an optional embodiment, a service program for executing the text-and-graphics mixed typesetting solution can be installed on the electronic device.

[0033] The embodiments of the present application can be applied to various graphic typesetting scenarios. For example, the conversion scenario of digital picture book files. Specifically, in this scenario, the digital picture book files corresponding to the physical publications need to be converted into mixed graphic and text resources in the interactive interface of the online learning software. In particular, in practical applications, the technical solution provided by the present application can be applied across platforms in the mixed graphic and text scenarios in the Android system and Apple's IOS operating system, greatly expanding the application scenarios. In addition, in order to enrich the learning experience, corresponding interactive functions will be added to the mixed graphic and text resources. For specific interactive functions, please refer to the embodiments below.

[0034] Figure 1 A flowchart of a method for mixing text and images provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0035] 101. Obtain the original graphic and text resources to be processed;

[0036] 102. Analyze the original graphic resources to obtain text resources and image resources to be processed;

[0037] 103. Determine text arrangement features corresponding to the text resource according to the layout type to which the text resource belongs;

[0038] 104. Perform mixed arrangement processing on text resources and image resources according to text arrangement characteristics, and display the processed mixed text and image resources in an interactive interface of the client, so as to provide interactive functions to users through the mixed text and image resources in the interactive interface.

[0039] It should be noted that mixed text and images is a typesetting method, specifically, it refers to the method of mixing text and images. In actual applications, text can be arranged around the image, embedded below the image, or float above the image. Depending on the content of the image, the text can be in paragraph form or in curve form.

[0040] In the embodiment of the present application, different from the text format of pure text or the image format of pure image, the original image resource is a rich text containing rich format attributes, such as a digital picture book file containing text, pictures, and layout. For another example, the original image resource can also be a scanned file of various physical publications, or other image resources containing multiple format attributes such as pictures and text.

[0041] In practical applications, the original picture book resources include picture book publication resources. Specifically, the picture book publication resources can be scanned files of paper picture books, or typeset picture book files provided by publishers or copyright holders, such as portable document format (PDF) files of picture books.

[0042] Based on the above introduction, an optional implementation method of obtaining the original graphic resources to be processed in 101 is to import the digital picture book files from the interface authorized by the publisher, for example, to obtain the digital picture book files corresponding to various courses through Adobe InDesign from the official interface authorized by Oxford. Alternatively, another optional implementation method may also be for relevant personnel to upload scanned copies of various teaching materials, such as scanned copies of illustrations, scanned copies of text, etc.

[0043] In order to restore the above original graphic resources in the client, it is necessary to split the images and texts in the original graphic resources, and rearrange the split images and texts to adapt to the interactive interface layout on the client side.

[0044] It is worth noting that the client involved in the embodiments of the present application may be an online learning software running on a terminal device, through which a user can implement an online learning scenario. For example, it may be an online language learning scenario, specifically, a picture book reading scenario, a Chinese word learning scenario, an English word learning scenario, a German word learning scenario, or a learning scenario of another type of language.

[0045] To achieve the above goal, the original graphic resources need to be split first. Specifically, in step 102, the original graphic resources can be parsed to obtain text resources and image resources to be processed.

[0046] In the embodiment of the present application, the text resource may be a text in any format, and the image resource may be an image in any format. In practical applications, the text in any format includes but is not limited to txt format, pdf format, doc format, docx format, and wps format, and the image in any format includes but is not limited to bmp format, jpg format, gif format, raw format, and WMF format.

[0047] For example, assuming that the original graphic resource is a digital picture book file exported by Adobe InDesign, in 102, the corresponding text resources and image resources can be identified by page from the digital picture book file, and the text resources and image resources corresponding to each page can be set in different layers.

[0048] Furthermore, after parsing out the text resources and image resources, it is necessary to determine the layout features of the text resources and image resources. Specifically, in 103, the text arrangement features corresponding to the text resources are determined according to the layout type to which the text resources belong. Optionally, the text resources and image resources, as well as related layout features such as text arrangement features, can also be converted into a JavaScript Object Notation (JSON) format configuration table recognizable by the client. In some scenarios, the configuration table can also be aligned with the audio file, and the JSON format configuration table corresponding to the audio file is generated according to the alignment result. Furthermore, the JSON format configuration tables corresponding to the graphic resources and audio files are uploaded to the server device in the form of encrypted compressed packages, so that the subsequent client can directly obtain these configuration tables for realizing the display of the above-mentioned graphic resources and audio files.

[0049] In practical applications, the layout features of image resources can be identified from the original graphic resources, such as image size, arrangement, etc. The layout features of text resources need to be specifically extracted according to the layout type to which the text resources belong.

[0050] Specifically, assuming that the layout type includes a paragraph type or a curve type, different feature extraction methods are required for text resources of different layout types to obtain corresponding text arrangement features, thereby providing a basis for subsequent text resource rendering in the client.

[0051] In 103, an optional implementation method of determining the text arrangement feature corresponding to the text resource according to the layout type to which the text resource belongs is as follows:

[0052] Determine whether the layout type of the text resource is a paragraph type or a curve type; if the text resource belongs to the paragraph type, use the feature calculation method corresponding to the paragraph type to calculate the paragraph features corresponding to the text resource; if the text resource belongs to the curve type, use the feature calculation method corresponding to the curve type to calculate the character features and curve features corresponding to the text resource.

[0053] In the above steps, there are multiple implementation methods for determining whether the layout type to which the text resource belongs is a paragraph type or a curve type. One implementation method is to identify the shape of the text resource and determine the layout type to which the text resource belongs based on the recognition result. For example, when the shape of the text resource is multiple lines, it can be determined that the layout type to which the text resource belongs is a paragraph type. For example, when the shape of the text resource is a single line, and there are multiple characters in the text resource with inconsistent orientations, it can be determined that the layout type to which the text resource belongs is a curve type.

[0054] The following describes how to extract layout features of text resources of different layout types.

[0055] For text resources of paragraph type, an optional implementation method of calculating the feature corresponding to the paragraph type is adopted in 103 to calculate the paragraph feature corresponding to the text resource, specifically: determining the paragraph layout information corresponding to each paragraph in the text resource and the attribute information corresponding to each character in the text resource according to the text resource; and using the paragraph layout information and the attribute information as paragraph features.

[0056] For example, the text boxes corresponding to each paragraph in the text resource are obtained, and the paragraph layout information corresponding to each paragraph is determined according to the attribute information of each text box and the text resources in each text box. In the embodiment of the present application, the paragraph layout information includes but is not limited to horizontal alignment, left inner margin, right inner margin, first line indent, vertical alignment, spacing after paragraph, spacing before paragraph, line spacing, and character height. Then, for the text resources in each text box, the attribute information corresponding to each character is parsed. In the embodiment of the present application, the attribute information corresponding to each character includes but is not limited to character size, character type, and character position (such as whether it is the beginning or the end of the line). In actual application, optionally, the above-mentioned paragraph layout information and the attribute information corresponding to each character can be used as paragraph features, stored in the JSON configuration table of the corresponding page according to the corresponding page identifier, and provide a basis for the paragraph text in the following text.

[0057] For a text resource of the curve type, an optional implementation method of using a feature calculation method corresponding to the paragraph type in 103 to calculate the paragraph feature corresponding to the text resource is as follows:

[0058] Determine the Bezier curve type according to the text resource; determine the key point information of the corresponding Bezier curve according to the Bezier curve type, and use the key point information as the curve feature; calculate the origin position and rotation angle corresponding to each character in the text resource according to the key point information and the Bezier curve formula, and use the origin position and rotation angle as the character feature.

[0059] Among them, the Bezier curve, also known as the Bezier curve or Bezier curve, is a mathematical curve applied to two-dimensional graphics applications. The Bezier curve is a very important parametric line in computer graphics. In the embodiment of the present application, the Bezier curve type includes but is not limited to a first-order Bezier curve, a second-order Bezier curve, and a third-order Bezier curve.

[0060] For example, determine the type of Bezier curve that matches the text resource, such as the first-order Bezier curve, the second-order Bezier curve, and the third-order Bezier curve. Then, determine the key point information of the Bezier curve based on the judgment result, such as the starting point, control point, and end point, and use these key point information as curve features and store them in the JSON configuration table of the corresponding page. You can also calculate the origin position and rotation angle corresponding to each character or punctuation in the text resource based on these key point information and the Bezier curve formula, and use these origin positions and rotation angles as the character features corresponding to each character or punctuation, and synchronously store them in the JSON configuration table of the corresponding page, providing a basis for the curve text in the following text. Figure 2 Take the character "Q" shown in the figure as an example. The origin coordinates of the character are Figure 2 The origin of the coordinate system.

[0061] Furthermore, after determining the layout features of the text resources and the image resources, in 104 , the text resources and the image resources may be mixed according to the text arrangement features, and the processed mixed text and image resources may be displayed in the interactive interface of the client.

[0062] Specifically, it is assumed that the image resource includes an image arrangement order. Based on this, an optional implementation method of mixing the text resource and the image resource according to the text arrangement feature in 104 is:

[0063] The position and orientation of each character in the text resource are determined according to the text arrangement characteristics; the text resource and the image resource are rendered according to the image arrangement order, the position and orientation of each character, so as to obtain a mixed text and image resource.

[0064] In an optional embodiment, if the text resource is of a curve type, the step of determining the position and orientation of each character in the text resource based on the text arrangement characteristics can be specifically implemented as follows: determining the trajectory parameters of the Bezier curve based on the curve characteristics; and calculating the position and orientation of each character in the text resource in the interactive interface based on the trajectory parameters and character characteristics.

[0065] Specifically, during the shuffling process, multiple characters in the text resource need to be arranged along the direction of the Bezier curve. Therefore, it is necessary to determine the Bezier curve type stored in the configuration table and the corresponding starting point, control point, and end point, and then select the first-order / second-order / third-order Bezier curve formula according to the Bezier curve type, and calculate the trajectory parameters of the Bezier curve through the selected formula. In actual applications, the Bezier curve can be set to be invisible to the user or visible to the user.

[0066] Furthermore, in 104, the trajectory parameters of the Bezier curve are used as the reference baseline, the left starting point of the reference baseline is used as the drawing origin, and the origin coordinates (i.e., the position) and the rotation angle (i.e., the orientation) of each character in the interactive interface are calculated by difference calculation based on the character features of each character stored in the configuration table.

[0067] Finally, in 104, after determining the trajectory parameters of the Bezier curve, the origin coordinates of each character, and the rotation angle, the text resource and the image resource are rendered in combination with the image arrangement order contained in the image resource and the glyph features supported by the client to obtain a mixed text and image resource. Specifically, in the process of rendering the text resource, the Bezier curve can be used as a reference line, the left starting point of the reference line can be used as the drawing origin, and each character can be drawn according to the origin coordinates and the rotation angle of each character.

[0068] Optionally, the text resource can be drawn on multiple Bezier curves. Figure 3 Taking the picture book learning scenario shown as an example, there are Bezier curves connected end to end in the current interactive interface. In this case, text resources can be drawn on the continuously connected Bezier curves. Specifically, in the process of drawing a character, if it is detected that the original coordinates of the current character have exceeded the length of the current Bezier curve, it is determined whether there is a next Bezier curve. If so, it is determined to draw the character in the next Bezier curve.

[0069] In another optional embodiment, if the text resource is of a paragraph type, the step of determining the position and orientation of each character in the text resource based on the text arrangement characteristics can be specifically implemented as follows: determining the first character of each line in the text resource through a regular expression; determining the position and orientation of the first character of each line in the interactive interface based on the paragraph characteristics.

[0070] In the above steps, the first character and the last character of each line in the text resource are identified by regular expressions. Then, it is determined whether the last character identified is a set character. Specifically, the set characters can be set according to the grammar rules. For example, upper double quotation marks and upper book title marks cannot appear at the end of each line, and lower double quotation marks and lower book title marks cannot appear at the beginning of each line. Assuming that the set characters are upper double quotation marks and upper book title marks, if the last character is the set character, the last character is set as the first character of the next line.

[0071] In addition to the above grammar rules, optionally, the text resources can also be typeset according to the characters in the digital picture book file. For example, assuming that the set characters are long spaces, short spaces, tabs, and line breaks, based on this, it can also be determined whether the set characters are the first characters. If the set characters are the first characters, the set characters are set to the end of the previous text line.

[0072] Finally, in step 104, after determining the position and orientation of the first character of each line in the interactive interface, the text resource and the image resource are rendered in combination with the image arrangement order contained in the image resource and the glyph features supported by the client to obtain a mixed text and image resource. Figure 4 Taking the picture book learning scenario shown as an example, in the process of rendering paragraph-type text resources, one or more characters can be used to form a line according to the rectangular area where the text resource is located, and then each character in each line is drawn according to the line spacing, paragraph spacing, character height, the position of the first character in each line, and the direction, so as to restore the paragraph-type text resources in the interactive interface.

[0073] In the related art, on the picture book display page, text is drawn in a specified rectangular area in units of paragraphs. However, this mixed text and image layout method cannot detect whether a word or phrase in a paragraph is hit, and it is difficult to accurately calculate the interaction position in the picture book reading scene, affecting the user's interactive experience.

[0074] In response to this technical problem, in an embodiment of the present application, optionally, the processed mixed text and image resources are displayed in the interactive interface in units of characters. Based on this, providing interactive functions to users through mixed text and image resources in the interactive interface can be implemented as follows: in response to a selection instruction for any character in the interactive interface, executing an interactive function corresponding to any character, and displaying the result of the interactive function execution. Among them, the interactive functions corresponding to any character include but are not limited to any one or combination of query, retrieval, reading aloud, explanation, practice push, and annotation. Therefore, by displaying mixed text and image resources in units of characters, it is possible to avoid the situation where it is impossible to determine whether a single hit or a phrase is hit, thereby effectively improving the detection accuracy of the interactive position and enhancing the user experience.

[0075] For example, in the English picture book reading scenario, the corresponding words or phrases in the picture book are highlighted in the interactive interface as the audio progresses, so that the user can know the current reading progress. In response to the user's single-click instruction on the interactive interface, the word corresponding to the single-click instruction is calculated according to the contact position, and the meaning of the word is queried, and the queried meaning is displayed in the current interactive interface.

[0076] Whether it is a curve type text resource or a paragraph type text resource, in the interactive interface, a single character is used as the smallest unit for drawing. Optionally, when drawing each character, the drawing parameters of each character can also be stored, such as Figure 2 The position coordinates, glyph width, ascent height (Ascent), and descent height (Descent) are shown in FIG. In the above example, the character corresponding to the single-click instruction can be calculated according to the contact position and the drawing parameters of each character, so as to accurately determine the hit character.

[0077] In this embodiment, the text arrangement features corresponding to the text resources are determined in a targeted manner by the layout type to which the text resources belong, so that the mixed typesetting of text resources and image resources can be more accurately realized according to the text arrangement features, thereby improving the rendering accuracy of the final mixed typesetting resources of images and texts, improving the efficiency of mixed typesetting of images and texts, and improving the user experience. In addition, by providing interactive functions to users through mixed typesetting resources of images and texts in the interactive interface, the user's interactive experience can be enriched and the interest in learning picture books can be enhanced.

[0078] It should be noted that the execution subject of each step of the method provided in the above embodiment can be the same device, or the method can be executed by different devices. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of steps 103 and 104 can be device B; and so on.

[0079] In addition, in some of the processes described in the above embodiments and the accompanying drawings, multiple operations appearing in a specific order are included, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or may be executed in parallel, and the sequence numbers of the operations, such as 101, 102, 103, 104, etc., are only used to distinguish different operations, and the sequence numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel.

[0080] It should be noted that the descriptions such as “first” and “second” in this article are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence, nor do they limit “first” and “second” to different types.

[0081] The following will describe in detail the image-text mixing device of one or more embodiments of the present application. Those skilled in the art will appreciate that these image-text mixing devices can be configured using commercially available hardware components through the steps taught in this solution.

[0082] In another embodiment of the present application, a device for mixing text and images is also provided. Figure 5 As shown, the device comprises:

[0083] Acquisition module 01, used to acquire original graphic resources to be processed, the original graphic resources including picture book publication resources;

[0084] Parsing module 02, used to parse the original graphic resources to obtain the text resources and image resources to be processed;

[0085] Determination module 03, used to determine the text arrangement feature corresponding to the text resource according to the layout type to which the text resource belongs;

[0086] The mixed display module 04 is used to mix text resources and image resources according to text arrangement characteristics, and display the processed mixed text and image resources in the interactive interface of the client, so as to provide interactive functions to users through the mixed text and image resources in the interactive interface.

[0087] Optionally, the layout type includes a paragraph type or a curve type.

[0088] Based on this, when the determination module 03 determines the text arrangement feature corresponding to the text resource according to the layout type to which the text resource belongs, it is specifically used to:

[0089] Determine whether the layout type to which the text resource belongs is the paragraph type or the curve type; if the text resource belongs to the paragraph type, use the feature calculation method corresponding to the paragraph type to calculate the paragraph feature corresponding to the text resource; if the text resource belongs to the curve type, use the feature calculation method corresponding to the curve type to calculate the character feature and curve feature corresponding to the text resource.

[0090] Optionally, the determining module 03 adopts the feature calculation method corresponding to the paragraph type to calculate the paragraph feature corresponding to the text resource, specifically for:

[0091] Determine paragraph layout information corresponding to each paragraph in the text resource and attribute information corresponding to each character according to the text resource; and use the paragraph layout information and the attribute information as the paragraph feature.

[0092] Optionally, the determining module 03 adopts the feature calculation method corresponding to the paragraph type to calculate the paragraph feature corresponding to the text resource, specifically for:

[0093] Determine a Bezier curve type according to the text resource, the Bezier curve type includes a first-order Bezier curve, a second-order Bezier curve, and a third-order Bezier curve; determine key point information of the corresponding Bezier curve according to the Bezier curve type, and use the key point information as the curve feature; calculate the origin position and rotation angle corresponding to each character in the text resource according to the key point information and the Bezier curve formula, and use the origin position and the rotation angle as the character feature.

[0094] Optionally, the image resource includes an order in which images are arranged.

[0095] Based on this, when the mixed arrangement display module 04 performs mixed arrangement processing on the text resource and the image resource according to the text arrangement feature, it is specifically used to:

[0096] The position and orientation of each character in the text resource are determined according to the text arrangement feature; and the text resource and the image resource are rendered according to the image arrangement order, the position and orientation of each character to obtain the mixed text and image resource.

[0097] Optionally, if the text resource is of a curve type, the mixed display module 04 determines the position and orientation of each character in the text resource according to the text arrangement feature, specifically for:

[0098] The trajectory parameters of the Bezier curve are determined according to the curve characteristics; and the position and orientation of each character in the text resource in the interactive interface are calculated according to the trajectory parameters and the character characteristics.

[0099] Optionally, if the text resource is of paragraph type, the mixed arrangement display module 04 determines the position and orientation of each character in the text resource according to the text arrangement feature, specifically for:

[0100] The first character of each line in the text resource is determined by a regular expression; and the position and orientation of the first character of each line in the interactive interface are determined according to paragraph features.

[0101] Optionally, when the mixed display module 04 determines the first character of each line in the text resource by using a regular expression, it is specifically used to:

[0102] Identify the first character and the last character of each line in the text resource through a regular expression; determine whether the identified last character is a set character, the set character includes upper double quotation marks and upper book title marks; if the last character is the set character, set the last character as the first character of the next line.

[0103] Optionally, the processed text-and-graphic mixed resources are displayed in the interactive interface in units of characters. Based on this, when the mixed display module 04 provides interactive functions to the user through the text-and-graphic mixed resources in the interactive interface, it is specifically used to: respond to a selection instruction for any character in the interactive interface, execute the interactive function corresponding to the any character, and display the execution result of the interactive function; wherein the interactive function is any one or a combination of query, retrieval, reading aloud, explanation, practice push, and annotation.

[0104] In another embodiment of the present application, an electronic device is provided, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0105] Memory, used to store computer programs;

[0106] The processor is used to implement the image and text mixing method described in the method embodiment when executing the program stored in the memory.

[0107] The communication bus 1140 mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 may be divided into an address bus, a data bus, a control bus, etc.

[0108] For ease of representation, Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0109] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0110] The memory 1130 may include a random access memory (RAM) or a non-volatile memory (non-volatile memory), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0111] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0112] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement each step that can be executed by an electronic device in the above method embodiment.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0117] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0118] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0119] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0120] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0121] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for mixed typesetting of images and texts, characterized in that: The method comprises: Acquire original graphic resources to be processed, wherein the original graphic resources include picture book publication resources; Parsing the original graphic resources to split the original graphic resources into text resources and image resources to be processed, wherein the text resources and image resources are arranged in different layers; Identify the shape of the text resource, and determine the layout type to which the text resource belongs according to the identification result, and use different feature extraction methods according to the layout type to which the text resource belongs to determine the text arrangement feature corresponding to the text resource; The text resource and the image resource are mixed according to the text arrangement feature, and the processed text-image mixed resource is displayed in the interactive interface of the client, so as to provide the user with interactive functions through the text-image mixed resource in the interactive interface.

2. The method for mixed typesetting of images and texts according to claim 1, characterized in that: The layout type includes a paragraph type or a curve type; The determining, according to the layout type to which the text resource belongs, the text arrangement feature corresponding to the text resource comprises: Determining that the layout type to which the text resource belongs is the paragraph type or the curve type; If the text resource belongs to the paragraph type, a feature calculation method corresponding to the paragraph type is used to calculate the paragraph feature corresponding to the text resource; If the text resource belongs to the curve type, a feature calculation method corresponding to the curve type is used to calculate character features and curve features corresponding to the text resource.

3. The method for mixed typesetting of images and texts according to claim 2, characterized in that: The adopting the feature calculation method corresponding to the paragraph type to calculate the paragraph feature corresponding to the text resource includes: Determine, according to the text resource, paragraph layout information corresponding to each paragraph in the text resource and attribute information corresponding to each character; The paragraph layout information and the attribute information are used as the paragraph features.

4. The method for mixed typesetting of images and texts according to claim 2, characterized in that: The adopting the feature calculation method corresponding to the curve type to calculate the character features and curve features corresponding to the text resource includes: Determine a Bezier curve type according to the text resource, where the Bezier curve type includes a first-order Bezier curve, a second-order Bezier curve, and a third-order Bezier curve; Determine key point information of the corresponding Bezier curve according to the Bezier curve type, and use the key point information as the curve feature; According to the key point information and the Bezier curve formula, the origin position and the rotation angle corresponding to each character in the text resource are calculated, and the origin position and the rotation angle are used as the character features.

5. The method for mixed typesetting of images and texts according to claim 1, characterized in that: The image resource includes an order of image arrangement; The mixing and sorting of the text resource and the image resource according to the text arrangement feature includes: Determining the position and orientation of each character in the text resource according to the text arrangement feature; The text resource and the image resource are rendered according to the image arrangement order, the position and orientation of each character, so as to obtain the mixed text and image resource.

6. The method for mixed typesetting of images and texts according to claim 5, characterized in that: If the text resource is of a curve type, determining the position and orientation of each character in the text resource according to the text arrangement feature includes: Determine the trajectory parameters of the Bezier curve according to the curve characteristics; The position and orientation of each character in the text resource in the interactive interface are calculated according to the trajectory parameters and character features.

7. The method for mixed typesetting of images and texts according to claim 5, characterized in that: If the text resource is of paragraph type, determining the position and orientation of each character in the text resource according to the text arrangement feature includes: Determine the first character of each line in the text resource by using a regular expression; The position and orientation of the first character of each line in the interactive interface are determined according to the paragraph features.

8. The method for mixed typesetting of images and texts according to claim 7, characterized in that: The determining the first character of each line in the text resource by using a regular expression includes: Identify the first character and the last character of each line in the text resource by using a regular expression; Determine whether the last character identified is a set character, wherein the set character includes an upper double quotation mark and an upper book title mark; If the last character is the set character, the last character is set as the first character of the next line.

9. The method for mixed typesetting of images and texts according to claim 1, characterized in that: Displaying the processed graphic and text mixed typesetting resources in the interactive interface in units of characters; Providing interactive functions to users through the mixed text and image resources in the interactive interface includes: In response to a selection instruction for any character in the interactive interface, executing an interactive function corresponding to the any character, and displaying an interactive function execution result; The interactive function is any one or a combination of query, retrieval, reading aloud, explanation, practice push, and annotation.

10. A device for mixing text and images, characterized in that: The device comprises: An acquisition module, used for acquiring original graphic resources to be processed, wherein the original graphic resources include picture book publication resources; A parsing module, used for parsing the original graphic resources to split the original graphic resources into text resources and image resources to be processed, wherein the text resources and image resources are arranged in different layers; A determination module, used to identify the shape of the text resource, and determine the layout type to which the text resource belongs according to the recognition result, and adopt different feature extraction methods according to the layout type to which the text resource belongs to determine the text arrangement feature corresponding to the text resource; The mixed display module is used to mix the text resources and the image resources according to the text arrangement characteristics, and display the processed mixed text and image resources in the interactive interface of the client, so as to provide interactive functions to users through the mixed text and image resources in the interactive interface.

11. An electronic device, characterized in that: include: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the image-text mixing method as described in any one of claims 1 to 9.

Images