Picture and text joint editing and correcting method and system for publications

Abstract

The invention discloses a picture and text joint editing and correcting method and system for publications, and belongs to the technical field of publication. The method comprises the steps that all pictures and picture names in the publications are read; matching the picture with the picture name; and taking the picture and the picture name which are matched with each other as input, taking related information of the publication as a prompt condition, and performing joint editing and correction on the picture and the picture name which are matched with each other through the multi-modal large model to obtain a joint editing and correction result. Compared with an existing method, the method has the advantages that the editing and correcting efficiency of the publication can be obviously improved, pictures and picture name contents in the publication can be edited and corrected in a combined mode, missing pictures and picture names in the publication can be supplemented, and the supplemented pictures and picture name contents can be edited and corrected in a combined mode. According to the method, the labor cost of publication editing work can be reduced, and harmful information can be automatically screened and prevented from flowing into the publishing market.

Description

Technical Field

[0001] The present invention relates to the field of editing and proofreading, and in particular to a method and system for jointly editing and proofreading pictures and texts of publications. Background Art

[0002] In the publishing field, it is often the case that the pictures and texts in the publications do not match, the pictures or texts are missing, the picture content does not match the corresponding theme of the text part well, or there are appropriate parts in the pictures or text parts.

[0003] Once these publications whose pictures and texts are inconsistent enter the market, they will have a very negative impact on the general readers, young students and even the society. Therefore, effective editing tools and methods are needed to edit and proofread the consistency of the content, style, artistic form, spiritual outlook and cultural expression of pictures and texts.

[0004] The current automated proofreading methods in the industry are mainly used to identify and organize scanned documents, and cannot intelligently proofread the above information of images and texts, for example: Document 1 (application number CN 201711097079.4) discloses a dynamic digital publishing system, which mainly includes a digital content semantic processing system, which is used to serve publishing companies and provide functions such as digital content semantic processing, domain ontology construction, and digital content editing and review. The digital publication editing and review system disclosed by the document is used to realize online editing, proofreading, and review of digital publication content to ensure the quality of digital publication content. It can only perform online editing, proofreading, and review of the content of the publication, and cannot accurately edit and proofread the attributes and matching of the images and corresponding text information.

[0005] Document 2 (application number CN 201810030535.1) discloses a text recognition editing and proofreading system that can be flexibly configured and defined, in which the computer engine module is used to automatically analyze the layout, image segmentation and image recognition of the image to be processed, and to "normalize" the recognition results of multiple recognition engines, and determine the text confidence of the editing system based on confidence comparison, and then revise the text confidence of the editing system based on online learning. However, the document processes images and texts independently, and there is no content matching editing and proofreading.

[0006] Document 3 (application number CN 201810493009.9) discloses a text recognition and editing method, which includes: performing image preprocessing on a document image; using a multi-channel recognition engine to recognize the text in the document after image preprocessing to obtain a text recognition result. However, this document does not solve the problem of the consistency of deep information between text and image.

[0007] Document 4 (application number CN 201910048711.9) discloses an auxiliary editing method and device, and a storage medium, which are mainly used to obtain multiple items to be processed in a file to be edited; each of the items to be processed is used as a main item, and reference information associated with the main item is retrieved from the file to be edited. However, this document does not have the function of image and text editing.

[0008] Document 5 (application number CN 202010526068.9) discloses a document editing and coordination solution, but does not involve the ability to jointly edit specific texts and images.

[0009] Document 6 (application number CN 202280019785.X) discloses a selective editing method for images, but the document does not involve consistent editing of images and texts.

[0010] From the above analysis, it can be seen that there are currently no methods and systems related to the joint editing of pictures and texts in the industry, and it is impossible to deal with the inconsistency problem of pictures and texts well. The joint editing of pictures and texts in publications mainly relies on manual review by editors, which is not only inefficient, but also easy to miss problematic pictures and texts due to human negligence, resulting in inappropriate content being published and distributed.

[0011] Therefore, it is urgent to develop a method and system for jointly editing and proofreading images and text content, through which joint editing and proofreading of images and text content is becoming an increasingly urgent need in the field of modern digital publishing. Summary of the invention

[0012] In view of the problem in the prior art that there is a lack of joint editing means for the images and text contents of publications, an object of the present invention is to provide a method and system for joint editing of images and texts of publications, so as to at least partially solve the above problem.

[0013] To achieve the above object, the technical solution of the present invention is: In a first aspect, the present invention provides a method for jointly editing pictures and texts of publications, the method comprising the following steps: Read all the pictures and figure titles in the publication; Pairing the picture with the picture title; Using mutually matching pictures and picture titles as input and relevant information of the publication as prompt conditions, the mutually matching pictures and picture titles are jointly edited and proofread through a multimodal large model to obtain a joint editing result.

[0014] In some preferred embodiments, the step of reading all pictures and picture titles in the publication also includes reading the positions of all pictures and the positions of all picture titles.

[0015] In some preferred embodiments, the step of pairing the picture and the picture title comprises: Extract feature vectors from each of the images and each of the image names using a multimodal feature extractor to obtain corresponding image vectors and image name vectors; For each of the pictures, respectively calculate the similarity distance between the picture vector corresponding to the picture and each picture name vector, and determine the minimum similarity distance. When the minimum similarity distance is less than the distance threshold, the picture name corresponding to the minimum similarity distance is determined to be a picture name that matches the picture, otherwise the picture is determined to be an isolated picture; or directly predict the matching relationship between the picture vector and the picture name vector through a deep learning classification model, so as to determine whether the picture and the picture name match. If the picture does not match all the picture names, it is determined to be an isolated picture; and / or, For each of the image names, the similarity distance between the image name vector corresponding to it and each picture vector is calculated respectively, and the minimum similarity distance is determined. When the minimum similarity distance is less than the distance threshold, the picture corresponding to the minimum similarity distance is determined to be a picture matching the image name, otherwise the image name is determined to be an isolated image name; or through a deep learning classification model, the matching relationship between the image vector and the image name vector is directly predicted to determine whether the picture and the image name match. If the image name does not match all the pictures, it is determined to be an isolated image name.

[0016] In some preferred embodiments, the method further comprises the following steps: For isolated images that cannot be paired, using the relevant information of the publication as a prompt condition, obtaining a supplementary image name that matches the isolated image through a generative model, and jointly editing the isolated image and the supplementary image name that match each other; and For isolated figure names that cannot be paired, the relevant information of the publication is used as a prompt condition, and a supplementary image matching the isolated figure name is obtained through a generative model, and the matching supplementary icons and the isolated figure name are jointly edited.

[0017] In some preferred embodiments, the method further comprises the following steps: If the isolated image and the supplementary figure name cannot pass the joint editing, the reason for not passing the joint editing will be merged into the relevant information of the publication, and this will be used as a new prompt condition to input the generation model again to obtain a new supplementary figure name, until the joint editing is passed or the preset number of generation times is reached; and, If the isolated image name and the supplementary image cannot pass the joint editing, the reason for failing the joint editing will be merged into the relevant information of the publication, and this will be used as a new prompt condition and input into the generation model again to obtain a new supplementary image, until the joint editing is passed or the preset number of generations is reached.

[0018] In some preferred embodiments, the mutually matching pictures and picture names and their joint editing results are saved and form editing records for manual verification, and the multimodal large model is fine-tuned according to the manual verification results.

[0019] In some preferred embodiments, the relevant information of the publication includes one or more of the name, summary, and target readership of the publication.

[0020] In some preferred embodiments, the multimodal large model jointly edits the matching pictures and picture titles to include one or more of the following: whether the pictures and picture titles are consistent in content, whether they are consistent in style or spirit, whether they are consistent in nationality, whether they are consistent in age, whether the content is suitable for readers' age, and whether there is no vulgar content.

[0021] In some preferred embodiments, the multimodal large model is internVL2, and the generative model is Tongyi Qianwen or Zhipu Qingyan.

[0022] In a second aspect, the present invention further provides a system for jointly editing and proofreading pictures and texts of publications, comprising: Image reading module, used to read all images in the publication; Figure name reading module, used to read all figure names in publications; A pairing module, used for pairing each of the pictures with each of the picture names; And an editing module, which is used to take the mutually matching pictures and picture names as input and the relevant information of the publication as prompt conditions, and jointly edit the mutually matching pictures and picture names through a multimodal large model to obtain a joint editing result.

[0023] By adopting the above technical solution, the beneficial effect of the present invention is that: compared with the existing method, the method of the present invention can significantly improve the editing and proofreading efficiency of publications, and can jointly edit and proofread the pictures and picture names in the publications, and can also supplement the missing pictures and picture names in the publications, and jointly edit and proofread the supplemented pictures and picture names. The method of the present invention can reduce the labor cost of editing and proofreading publications, and can automatically screen to prevent harmful information from entering the publishing market. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1It is a schematic diagram of the flow of the method for joint editing of pictures and texts for publications in the present invention.

[0025] Figure 2 It is a schematic diagram of the overall process of the present invention.

[0026] Figure 3 This is the reference code for reading pictures in documents through the PyPDF library in the present invention.

[0027] Figure 4 The figure is a schematic diagram of the process of reading the image names in the document by using the large language model in the present invention.

[0028] Figure 5 It is a schematic diagram of respectively extracting picture vectors and picture name vectors through CLIP feature extractor in the present invention.

[0029] Figure 6 This is a schematic diagram of matching pictures and picture names in the present invention.

[0030] Figure 7 This is a schematic diagram of a picture related to the key point recognition of human hand gestures in the present invention.

[0031] Figure 8 This is a schematic diagram of the joint editing of images and image titles using a multimodal large model in the present invention.

[0032] Fig. 9 It is a schematic diagram of the structure of the picture and text joint editing system used for publications in the present invention. DETAILED DESCRIPTION

[0033] The specific embodiments of the present invention are further described below in conjunction with the accompanying drawings. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. Example 1

[0034] A method for jointly editing pictures and texts for publications, such as Figure 1-2 As shown, the method comprises the following steps: S1. Read all figures and figure titles in the publication.

[0035] S2. Pair the pictures and their titles.

[0036] S3. Take the matching pictures and picture titles as input, and the relevant information of the publication as prompt conditions, and jointly edit the matching pictures and picture titles through a multimodal large model to obtain the joint editing results.

[0037] In step S1, all images in the publication are read through a document processing library, such as the PyPDF library, and the location of the image is obtained while reading the image. Figure 3 As shown in the figure, it is the code used to extract pictures from publications through the PyPDF library. Based on the pictures that have been determined, the publications are analyzed through the large language model, and the text content corresponding to the pictures is identified from the publications, that is, the picture name is read, and the position of the picture name is also obtained while reading the picture name. The large language model can be the KIMI artificial intelligence assistant. Figure 4 As shown in the figure, the result of processing the first chapter of the "Computer Vision Textbook" through the KIMI artificial intelligence assistant. After verification, there is no error in the read image name information.

[0038] In step S2, the specific steps of pairing the picture and the picture title include: S21. Extract feature vectors for each image and each image name through a multimodal feature extractor to obtain corresponding image vectors and image name vectors.

[0039] S22. For each picture, respectively calculate the similarity distance between the corresponding picture vector and each picture name vector, and determine the minimum similarity distance. When the minimum similarity distance is less than the distance threshold, the picture name corresponding to the minimum similarity distance is determined to be the picture name that matches the picture, otherwise the picture is determined to be an isolated picture.

[0040] The multimodal feature extractor is configured as a CLIP feature extractor, such as the CLIP feature extractor disclosed in the document “Radford A, Kim JW, Hallacy C, et al. Learning transferable visual models from natural language supervision [C] / / International conference on machine learning. PMLR, 2021:8748-8763.”

[0041] like Figure 5 As shown, the CLIP feature extractor can extract feature vectors from images to obtain image vectors, and can also extract feature vectors from image names to obtain image name vectors. Moreover, the feature vectors extracted from images and image names with matching contents are very close.

[0042] like Figure 6As shown in the figure, the similarity distance between each image vector and each image name vector is calculated. For a certain image, the image name with the smallest similarity distance to the image is determined. Then, it is determined whether the similarity distance is less than the distance threshold. If it is less than or equal to the distance threshold, it is considered that there is a matching relationship between the two. If it is greater than the distance threshold, it is considered that there is no image name matching the image, and the image is called an isolated image.

[0043] On the contrary, for a certain picture name, the similarity distance between the corresponding picture name vector and each picture vector is calculated respectively, and the minimum similarity distance is determined. When the minimum similarity distance is less than the distance threshold, the picture corresponding to the minimum similarity distance is determined to be the picture matching the picture name, otherwise the picture name is determined to be an isolated picture name.

[0044] It is easy to understand that the judgment of whether the image and the name match can also be carried out in the following way: that is, through a deep learning classification model, the matching relationship between the image vector and the name vector is directly predicted to determine whether the image and the name match. If the name does not match all the images, it is determined to be an isolated name; correspondingly, through a deep learning classification model, the matching relationship between the image vector and the name vector is directly predicted to determine whether the image and the name match. If the name does not match all the images, it is determined to be an isolated name. The above-mentioned deep learning classification model is, for example, the VNLI model disclosed in the article "What You See is What You Read? Improving Text-Image Alignment Evaluation" published by Michal Yarom et al. at the 37th Conference on Neural Information Processing Systems (NeurIPS 2023). Text or images are directly input into the model, and the model can predict whether they match. This is actually a classification task to determine whether the input text or image belongs to the same category, that is, a classification model based on deep learning.

[0045] After the processing of step S2, several groups of mutually matching pictures and picture names can be obtained.

[0046] In step S3, the multimodal big model is configured as internVL2, and the relevant information of the publication as the prompt condition (also called prompt word) includes one or more of the publication name, summary, and target reader group. The multimodal big model jointly edits each group of matching pictures and picture names to include: whether the pictures and picture names are consistent in content, whether they are consistent in style or spirit, whether they are consistent in nationality, whether they are consistent in age, whether the content is suitable for readers' age, and whether there is no vulgar content.

[0047] like Figure 7 As shown in the figure, it is a picture related to the recognition of key points of human gestures. It is input into the multimodal large model and given the picture name information "cat and dog classification". Then the name of the publication is set as "computer vision" and the target reader group is set as "higher vocational and technical college textbooks". Then joint editing is carried out, and the content of the joint editing is "Do the content of the picture and the picture name match?" The joint editing result given by the multimodal large model is "The content of the picture and the picture name does not match", and the picture analysis is given "The picture shows a human hand with marked points on the fingers and palm; the hand posture is a thumbs-up, and the other fingers are bent, similar to common gestures", such as Figure 8 shown.

[0048] Usually, after the joint editing results are given in step S3, these matching pictures and picture names and their joint editing results will be saved to form editing records. These editing records can be used for manual verification, and then the multimodal large model is fine-tuned according to the manual verification results to make it more accurate. Example 2

[0049] It is easy to understand that in step S2, for isolated pictures that cannot be paired, the method provided in the embodiment of the present invention also uses the relevant information of the publication as a prompt condition, and processes the isolated picture through the generation model to obtain a supplementary picture name that matches the isolated picture, and then jointly edits the matching isolated pictures and supplementary picture names through the content of step S3.

[0050] Similarly, for isolated image names that cannot be paired, the relevant information of the publication is used as a prompt condition, and a supplementary image matching the isolated image name is obtained through the generation model. Then, the matching supplementary icons and the isolated image name are jointly edited through the content of step S3.

[0051] For example, if the generative model is configured as Tongyi Qianwen or Zhipu Qingyan, the isolated image name is set as "Hand gesture recognition key point detection", and the relevant information of the publication is set as "Computer vision textbook for higher vocational colleges", then the generative model can generate the following Figure 7 Supplementary images shown.

[0052] It is easy to understand that although the generated model has been processed, it does not mean that the matching isolated pictures and supplementary figure names, or the matching isolated figure names and supplementary pictures can pass the joint editing. Therefore, subsequent processing is still required, including: If the isolated image and the corresponding supplementary figure name cannot pass the joint editing, the reason for not passing the joint editing will be merged into the relevant information of the publication, and this will be used as a new prompt condition to input the generation model again to obtain a new supplementary figure name, until it passes the joint editing or reaches the preset number of generation times; Similarly, if the isolated image name and the corresponding supplementary image cannot pass the joint editing, the reason for not passing the joint editing will be merged into the relevant information of the publication, and this will be used as a new prompt condition and input into the generation model again to obtain a new supplementary image, until it passes the joint editing or reaches the preset number of generations.

[0053] In addition, when the preset number of generation times is reached and the joint editing still cannot be passed, an alarm is usually generated to facilitate manual intervention. Manual intervention refers to manual inspection and analysis of problems, manual editing, and recording of relevant information. Example 3

[0054] A system for joint editing of images and texts for publications, such as Fig. 9 As shown, the system includes a picture reading module, a picture name reading module, a pairing module and an editing module.

[0055] The image reading module is used to read all images in a publication, for example, through a document processing library, and can also determine the location of the images. For example, the document processing library is the PyPDF library.

[0056] The figure name reading module is used to read all the figure names in the publication, for example, by analyzing the publication through a large language model, identifying and reading the figure names in the publication, and also determining the location of the figure names. For example, the large language model can be the KIMI artificial intelligence assistant.

[0057] The pairing module is used to pair each picture with each picture name. For example, the CLIP feature extractor extracts feature vectors for each picture and each picture name through a multimodal feature extractor to obtain corresponding picture vectors and picture name vectors. Then, for each picture, the similarity distance between the picture vector corresponding to it and each picture name vector is calculated, and the minimum similarity distance is determined. When the minimum similarity distance is less than the distance threshold, the picture name corresponding to the minimum similarity distance is determined to be the picture name that matches the picture, otherwise the picture is determined to be an isolated picture. The same is true for the picture name.

[0058] The editing and proofreading module is used to take matching pictures and picture titles as input, and use the relevant information of the publication as prompt conditions, and jointly edit the matching pictures and picture titles through a multimodal large model to obtain the joint editing and proofreading results.

[0059] The multimodal big model is configured as internVL2. The relevant information of the publication as the prompt condition (also called prompt word) includes one or more of the publication name, summary, and target reader group. The multimodal big model jointly edits the matching pictures and picture names in each group, including whether the pictures and picture names are consistent in content, style or spirit, nationality, age, content suitable for readers, and whether there is no vulgar content.

[0060] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions and variations of these embodiments are made without departing from the principles and spirit of the present invention, and still fall within the scope of protection of the present invention.

Claims

1. A method for jointly editing pictures and texts of publications, characterized in that: The method comprises the following steps: Read all the pictures and figure titles in the publication; Pairing the picture with the picture title; Using mutually matching pictures and picture titles as input and relevant information of the publication as prompt conditions, the mutually matching pictures and picture titles are jointly edited and proofread through a multimodal large model to obtain a joint editing result.

2. The method according to claim 1, characterized in that: The step of reading all pictures and picture names in the publication also includes reading the positions of all pictures and the positions of all picture names.

3. The method according to claim 2, characterized in that: The step of pairing the picture and the picture name comprises: Extract feature vectors from each of the images and each of the image names using a multimodal feature extractor to obtain corresponding image vectors and image name vectors; For each of the pictures, respectively calculate the similarity distance between the picture vector corresponding to it and each picture name vector, and determine the minimum similarity distance. When the minimum similarity distance is less than the similarity distance threshold, it is determined that the picture name corresponding to the minimum similarity distance is a picture name that matches the picture, otherwise it is determined that the picture is an isolated picture; or through a deep learning classification model, directly predict the matching relationship between the picture vector and the picture name vector, so as to determine whether the picture and the picture name match, if the picture does not match all the picture names, it is determined to be an isolated picture; and / or, For each of the image names, the similarity distance between the image name vector corresponding to it and each picture vector is calculated respectively, and the minimum similarity distance is determined. When the minimum similarity distance is less than the similarity distance threshold, the picture corresponding to the minimum similarity distance is determined to be a picture matching the image name, otherwise the image name is determined to be an isolated image name; or through a deep learning classification model, the matching relationship between the image vector and the image name vector is directly predicted to determine whether the picture and the image name match. If the image name does not match all the pictures, it is determined to be an isolated image name.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises the following steps: For isolated images that cannot be paired, using the relevant information of the publication as a prompt condition, obtaining a supplementary image name that matches the isolated image through a generative model, and jointly editing the isolated image and the supplementary image name that match each other; and For isolated figure names that cannot be paired, the relevant information of the publication is used as a prompt condition, and a supplementary image matching the isolated figure name is obtained through a generative model, and the matching supplementary icons and the isolated figure name are jointly edited.

5. The method according to claim 4, characterized in that: The method further comprises the following steps: If the isolated image and the supplementary figure name cannot pass the joint editing, the reason for not passing the joint editing will be merged into the relevant information of the publication, and this will be used as a new prompt condition to input the generation model again to obtain a new supplementary figure name, until the joint editing is passed or the preset number of generation times is reached; and, If the isolated image name and the supplementary image cannot pass the joint editing, the reason for failing the joint editing will be merged into the relevant information of the publication, and this will be used as a new prompt condition and input into the generation model again to obtain a new supplementary image, until the joint editing is passed or the preset number of generations is reached.

6. The method according to claim 1, characterized in that: The mutually matching pictures and picture names and their joint editing results are saved and form editing records for manual verification, and the multimodal large model is fine-tuned according to the manual verification results.

7. The method according to claim 1, characterized in that: The relevant information of the publication includes one or more of the name, summary, and target readership of the publication.

8. The method according to claim 1, characterized in that: The multimodal large model jointly edits the matching pictures and picture titles to include one or more of the following: whether the pictures and picture titles are consistent in content, whether they are consistent in style or spirit, whether they are consistent in nationality, whether they are consistent in age, whether the content is suitable for readers' age, and whether there is no vulgar content.

9. The method according to claim 1, characterized in that: The multimodal large model is internVL2, and the generation model is Tongyi Qianwen or Zhipu Qingyan.

10. A system for joint editing of pictures and texts for publications, characterized in that: include: Image reading module, used to read all images in the publication; Figure name reading module, used to read all figure names in publications; A pairing module, used for pairing each of the pictures with each of the picture names; And an editing module, which is used to take the mutually matching pictures and picture names as input and the relevant information of the publication as prompt conditions, and jointly edit the mutually matching pictures and picture names through a multimodal large model to obtain a joint editing result.

Images