Image-text mixed arrangement method and system based on large language model, medium and equipment

Abstract

The invention provides an image-text mixed arrangement method and system based on a large language model, a medium and equipment, and the method comprises the steps: S1, positioning an image in a target document, obtaining a text description associated with the image, and generating a text annotation corresponding to the image according to the text description associated with the image in the target document by using the large language model; s2, performing document retrieval on the target document by using the large language model to obtain corresponding retrieval characters, performing relevance processing on the obtained retrieval characters and character annotations corresponding to the pictures, and when the retrieval characters and the character annotations corresponding to the pictures are semantically associated, adding the current picture into the context of the retrieval characters, so as to obtain a retrieval result; and image-text mixed arrangement is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a method, system, medium and device for mixing text and images based on a large language model. Background Art

[0002] Currently, the document retrieval based on the large language model only generates text results, and does not support the mixed arrangement of retrieved text and retrieved images.

[0003] The direct reason for not supporting the mixed arrangement of search text and search images is that the search text is no longer the original text after being generated and processed by the large language model, resulting in the destruction of the association between the original text and the original image by the generation of the large language model. At the same time, the original image in the file has at most one image title, and there are not enough text annotations to explain the "semantics" of the image in detail. The large language model mainly processes and generates text data, and does not understand the "semantics" of images, so it is inevitable that the image cannot be inserted into the context of the generated text. Summary of the invention

[0004] In view of the defects in the prior art, the purpose of the present invention is to provide a method, system, medium and device for mixing text and images based on a large language model.

[0005] A method for mixing text and images based on a large language model provided by the present invention includes:

[0006] Step S1: locate the image in the target document, obtain the text description associated with the image, and use the large language model to generate a text annotation corresponding to the image according to the text description associated with the image in the target document;

[0007] Step S2: Use the large language model to perform document retrieval on the target document to obtain the corresponding search text, and perform association processing on the obtained search text and the text annotation corresponding to the picture. When the search text and the text annotation corresponding to the picture are semantically associated, the current picture is added to the context of the search text to achieve mixed text and picture typesetting.

[0008] Preferably, the step S1 adopts:

[0009] Step S1.1: Obtaining image and text information in the target document according to document parsing technology;

[0010] Step S1.2: Obtain text content related to the image based on the reference to the image in the target document and the text on the image;

[0011] Step S1.3: The large language model obtains text annotations matching the current picture based on the acquired text content related to the picture.

[0012] Preferably, the step S2 adopts: using a large language model to calculate the text relevance between the search text and the text annotation corresponding to the picture, when the calculated text relevance meets the preset requirements, it is considered that the current search text and the text annotation corresponding to the picture are semantically associated.

[0013] Preferably, the use of a large language model to calculate the text relevance between the search text and the text annotation corresponding to the picture includes:

[0014] The large language model encodes the retrieved text and the text annotation corresponding to the image into vector representations, and then calculates the similarity between the two texts based on the vector representations of the text.

[0015] According to the present invention, a large language model-based image-text typesetting system includes:

[0016] Module M1: locate the image in the target document, obtain the text description associated with the image, and use the large language model to generate the text annotation corresponding to the image based on the text description associated with the image in the target document;

[0017] Module M2: Use the large language model to perform document retrieval on the target document to obtain the corresponding search text, and perform association processing on the obtained search text and the text annotation corresponding to the picture. When the search text and the text annotation corresponding to the picture are semantically related, the current picture is added to the context of the search text to achieve mixed text and picture typesetting.

[0018] Preferably, the module M1 adopts:

[0019] Module M1.1: Obtain image and text information in the target document based on document parsing technology;

[0020] Module M1.2: Obtain text content related to the image based on the target document's reference to the image and the text on the image;

[0021] Module M1.3: The large language model obtains text annotations that match the current image based on the acquired text content related to the image.

[0022] Preferably, the module M2 adopts: using a large language model to calculate the text relevance between the search text and the text annotation corresponding to the picture, when the calculated text relevance meets the preset requirements, it is considered that the current search text and the text annotation corresponding to the picture are semantically related.

[0023] Preferably, the use of a large language model to calculate the text relevance between the search text and the text annotation corresponding to the picture includes:

[0024] The large language model encodes the retrieved text and the text annotation corresponding to the image into vector representations, and then calculates the similarity between the two texts based on the vector representations of the text.

[0025] According to an electronic device provided by the present invention, the electronic device includes a memory and at least one processor, and instructions are stored in the memory;

[0026] The at least one processor calls the instructions in the memory to enable the electronic device to execute the various steps of the image-text mixing method based on the large language model as described above.

[0027] According to a computer-readable storage medium provided by the present invention, instructions are stored on the computer-readable storage medium, and when the instructions are executed by a processor, the various steps of the above-mentioned image-text mixing method based on a large language model are implemented.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] 1. The present invention supports the generation of mixed text and image search, which greatly increases the amount of information generated by the search and improves the user experience;

[0030] 2. The present invention generates new text annotations from the relevant text associated with the original picture, which is a kind of information enhancement, and the "semantics" expressed by it far exceeds the original picture number and picture title;

[0031] 3. Pictures and texts are associated through "semantics". Even if the original text is regenerated, as long as the semantics are similar, it can still be associated with the corresponding picture. The association is very strong. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0033] Figure 1 The figure is a flow chart of the image-text typesetting method based on a large language model. DETAILED DESCRIPTION

[0034] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0035] Example 1

[0036] According to a method for mixing text and images based on a large language model provided by the present invention, Figure 1 As shown, including:

[0037] Step 1: The large language model generates a complete text annotation based on the original title of the image in the file and the original text that references the image, which serves as the "semantics" of the image.

[0038] Step 2: For the RAG (retrieval-augmented generation) task, after the large language model generates text, the large language model retrieves the images of the file and the text annotations corresponding to the images. If there is a semantic association between the generated text and the text annotations corresponding to the images, the image is added to the text context generated by the large language model to achieve mixed text and image typesetting.

[0039] Specifically, the step 1 includes:

[0040] In the file, each picture has a corresponding picture number and picture title;

[0041] Use document parsing technology and rules to locate images in the target document and obtain text descriptions associated with the images;

[0042] According to the reference to the image in the file, find the complete context of the reference to the image in the file; for example, the image number is mentioned in the original text, such as: as shown in Figure x, or the above or below image is mentioned in the original text; or when there is no obvious reference information, the text before and after the image appears is considered to be the context of the reference to the image;

[0043] In addition to citing the context of the image, the text information in the image is also obtained through OCR technology. At the same time, the context, image title and text in the image are used as input information for the large model to generate annotations. Through the rich knowledge and common sense reasoning ability of the large model, the image can be accurately annotated. The text annotation not only has complete "semantics" but also has uniqueness. The text annotation is used as the text annotation of the image;

[0044] Specifically, the step 2 includes:

[0045] Based on the user's search requirements, the large language model generates a new paragraph of text; the newly generated text paragraph and the text annotation corresponding to the image are encoded into vector representations with specific semantics and features. In this process, the vocabulary, grammar, semantics, and context of the text are fully considered to ensure that the vector can accurately capture and represent the intrinsic meaning and structure of the text.

[0046] Then, based on the vector representation of the text, the large language model uses mathematical calculation methods to deeply analyze and measure the similarity between the two texts. This calculation comprehensively considers many key factors such as the degree of vocabulary matching, semantic consistency, concept relevance, and overall logical coherence, so as to obtain a comprehensive, accurate and practical similarity evaluation result.

[0047] If the text annotation corresponding to the image is "semantically" related to the newly generated text paragraph, that is, the similarity between the two paragraphs is high, the image will be inserted into the context of the newly generated text paragraph to form a mixed text and image layout.

[0048] The present invention also provides a large language model-based image-text typesetting system, which can be implemented by executing the process steps of the large language model-based image-text typesetting method, that is, those skilled in the art can understand the large language model-based image-text typesetting method as a preferred implementation of the large language model-based image-text typesetting system.

[0049] Example 2

[0050] Embodiment 2 is a preferred embodiment of Embodiment 1

[0051] A method for mixing text and images based on a large language model provided by the present invention includes:

[0052] Step 1: Get image annotations

[0053] Question: "Given the image's title and the full context in which it is cited, what is this image about? Give a very concise summary"

[0054] Answer: "Title": "STEP 7 Wizard: Select programming language and symbol blocks";

[0055] "Comment": "The image shows the process of selecting different programming languages ​​(such as ladder diagram, statement list or function block diagram) and symbol blocks in the STEP 7 programming environment for configuring and executing PLC programs."

[0056] Step 2: Get a new text according to the user's search requirements;

[0057] Question: Referring to the STEP7V53 Getting Started Manual, how should I select the programming language when creating a new project using the STEP 7 wizard? What programming languages ​​are there and the steps to select a programming language?

[0058] Answer: When creating a new project using the STEP 7 wizard, the steps for selecting the programming language are as follows:

[0059] 1. Start the STEP 7 software and open the wizard.

[0060] 2. At the corresponding step of the wizard, you will be prompted to select a programming language.

[0061] 3. Available programming languages ​​include:

[0062] -Ladder Logic (LAD): For users from the electrical engineering industry.

[0063] -Statement List (STL): Suitable for users coming from the field of computer technology.

[0064] -Function Block Diagram(FBD): Suitable for users coming from the field of circuit engineering.

[0065] 4. Choose the most appropriate programming language based on your background and project requirements.

[0066] 5. After selecting a programming language, you will usually need to confirm your choice before continuing to the next step.

[0067] The specific steps may involve the following operations:

[0068] -In the wizard interface, select "Select one of the programming languages: LadderLogic(LAD),Statement List(STL),or Function Block Diagram(FBD)".

[0069] -Click on the programming language of your choice.

[0070] -Click the "Next" or "Confirm" button to save your selection.

[0071] After selecting the programming language, you can start creating your program in OB1 (Organization Block 1), which is the highest programming level in an S7 program and is responsible for organizing the other program blocks. If necessary, you can change the programming language at a later date.

[0072] Step 3: Calculate the correlation between the image and the question;

[0073] Calculate the semantic relevance between <question & answer> and <image title & summary>, and filter based on the relevance score;

[0074] {'image link':'f824a852f14b92c0951f2782ef581e40.jpg','similarity score':10,'description':'"Title":"STEP 7 Wizard: Selecting Programming Language and Symbolic Blocks","Comment":"The image shows the process of selecting different programming languages ​​(such as Ladder Diagram, Statement List or Function Block Diagram) and symbolic blocks in the STEP 7 programming environment for configuring and executing PLC programs."'}

[0075] {'Image link':'0a97890f14736a360279f1bbcd02d25b.jpg','Similarity score':9,'description':'"Title":"Programming language selection wizard","Comment":"Image shows the programming language selection steps in the STEP 7 programming environment, allowing the user to select Ladder Logic, Statement List or Function Block Diagram for programming and confirm the settings to continue."'}

[0076] Step 4: Display the mixed text and image effect.

[0077] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0078] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A method for mixing text and images based on a large language model, characterized in that: include: Step S1: locate the image in the target document, obtain the text description associated with the image, and use the large language model to generate a text annotation corresponding to the image according to the text description associated with the image in the target document; Step S2: Use the large language model to perform document retrieval on the target document to obtain the corresponding search text, and perform association processing on the obtained search text and the text annotation corresponding to the picture. When the search text and the text annotation corresponding to the picture are semantically associated, the current picture is added to the context of the search text to achieve mixed text and picture typesetting.

2. The image-text typesetting method based on a large language model according to claim 1, characterized in that: The step S1 adopts: Step S1.1: Obtaining image and text information in the target document according to document parsing technology; Step S1.2: Obtain text content related to the image based on the reference to the image in the target document and the text on the image; Step S1.3: The large language model obtains text annotations matching the current picture based on the acquired text content related to the picture.

3. The image-text typesetting method based on a large language model according to claim 1, characterized in that: The step S2 adopts: using a large language model to calculate the text relevance between the search text and the text annotation corresponding to the picture. When the calculated text relevance meets the preset requirements, it is considered that the current search text and the text annotation corresponding to the picture are semantically related.

4. The image-text typesetting method based on a large language model according to claim 3, characterized in that: The method of calculating the text relevance between the search text and the text annotation corresponding to the picture by using the large language model includes: The large language model encodes the retrieved text and the text annotation corresponding to the image into vector representations, and then calculates the similarity between the two texts based on the vector representations of the text.

5. A text-image typesetting system based on a large language model, characterized in that: include: Module M1: locate the image in the target document, obtain the text description associated with the image, and use the large language model to generate the text annotation corresponding to the image based on the text description associated with the image in the target document; Module M2: Use the large language model to perform document retrieval on the target document to obtain the corresponding search text, and perform association processing on the obtained search text and the text annotation corresponding to the picture. When the search text and the text annotation corresponding to the picture are semantically related, the current picture is added to the context of the search text to achieve mixed text and picture typesetting.

6. The image-text typesetting system based on a large language model according to claim 5, characterized in that: The module M1 adopts: Module M1.1: Obtain image and text information in the target document based on document parsing technology; Module M1.2: Obtain text content related to the image based on the target document's reference to the image and the text on the image; Module M1.3: The large language model obtains text annotations that match the current image based on the acquired text content related to the image.

7. The image-text typesetting system based on a large language model according to claim 5, characterized in that: The module M2 adopts: using a large language model to calculate the text relevance between the search text and the text annotation corresponding to the picture. When the calculated text relevance meets the preset requirements, it is considered that the current search text and the text annotation corresponding to the picture are semantically related.

8. The image-text typesetting system based on a large language model according to claim 7, characterized in that: The method of calculating the text relevance between the search text and the text annotation corresponding to the picture by using the large language model includes: The large language model encodes the retrieved text and the text annotation corresponding to the image into vector representations, and then calculates the similarity between the two texts based on the vector representations of the text.

9. An electronic device, comprising a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory to enable the electronic device to execute the various steps of the image-text mixing method based on a large language model as described in any one of claims 1 to 4.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the image-text mixing method based on a large language model as described in any one of claims 1 to 4 are implemented.

Images