Image processing method, image generation method, electronic equipment and storage medium

Through multimodal model processing and fusion technology, scene text images with extremely small differences from real-world background images are generated, which solves the problem of large differences between generated images and real-world background images in the existing technology, and improves the subsequent processing effect.

CN118941674APending Publication Date: 2024-11-12ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410918082.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-09
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The scene text images generated in the prior art differ greatly from the real world background images, resulting in poor subsequent processing effects.

Method used

Use a multimodal model to obtain the image of the text to be rendered, obtain key points and generate text through text generation, process the target text area and target generated text, and fuse it with the image of the text to be rendered to generate the target image with very small differences .

Benefits of technology

Improve the accuracy of image processing and subsequent processing effects, ensure that the difference between the generated image and the original image is extremely small, and improve the processing effect.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941674A_ABST
    Figure CN118941674A_ABST
Patent Text Reader

Abstract

The invention discloses an image processing method, an image generation method, electronic equipment and a storage medium, and relates to the field of image processing in a large model technology. The method comprises the steps of obtaining an image of a to-be-rendered text; performing text generation based on the image of the to-be-rendered text by using a multi-modal model to obtain a plurality of key points and generated texts corresponding to the plurality of key points; processing the plurality of key points and the generated texts corresponding to the plurality of key points by using a multi-modal model to obtain a target text region and a target generated text corresponding to the target text region; and based on the target text area, fusing the target generated text with an image of the to-be-rendered text to obtain a target image. According to the method and the device, the technical problem that the processing effect of performing subsequent processing on the original image based on the generated image becomes poor due to the fact that the difference between the generated image after the text is generated and the original image is large in the related technology is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing in large model technology, and specifically, to an image processing method, an image generating method, an electronic device and a storage medium. Background Art

[0002] Scene text is one of the most challenging research objects in the field of visual text research. Research in this field has always been constrained by the lack of real-world data. Synthesizing visual text on real-world background images is the solution for current work. At the semantic level, in the process of generating scene text, although various methods can be used to find areas with consistent background image texture in real-world background images, and then generate scene text in these areas, there are still cases where scene text is generated in areas such as the sky where text should not appear. In addition, the generated text content is often randomly selected from the corpus, which is often irrelevant to the semantics of the real-world background image, which is also not in line with the real world. These will lead to a large difference between the generated image and the real image, and such generated images with large differences from the real image will cause data offset when used for downstream tasks, thereby affecting the effect on downstream tasks.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an image processing method, an image generating method, an electronic device and a storage medium, so as to at least solve the technical problem in the related art that the processing effect of subsequent processing of the original image based on the generated image is deteriorated due to the large difference between the generated image after generating text and the original image.

[0005] According to one aspect of an embodiment of the present application, there is provided an image processing method, comprising: obtaining an image of text to be rendered; generating text based on the image of the text to be rendered using a multimodal model to obtain a plurality of key points and generated texts corresponding to the plurality of key points, wherein the plurality of key points are used to represent positions in the image of the text to be rendered where the corresponding generated texts need to be rendered; processing the plurality of key points and the generated texts corresponding to the plurality of key points using the multimodal model to obtain a target text area and target generated texts corresponding to the target text area, wherein the target text area is used to represent an area in the image of the text to be rendered where the target generated text needs to be rendered; and fusing the target generated text with the image of the text to be rendered based on the target text area to obtain a target image.

[0006] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: capturing a scene image in a real scene; using a multimodal model to generate text based on the scene image to obtain multiple key points and scene texts corresponding to the multiple key points, wherein the key points are used to characterize the positions of the corresponding scene texts to be generated in the scene image; using the multimodal model to process the multiple key points and the scene texts corresponding to the multiple key points to obtain a target text area and a target scene text corresponding to the target text area, wherein the target text area is used to characterize the area in the scene image where the target scene text is to be generated; based on the target text area, the target scene text is fused with the scene image to obtain a scene text image.

[0007] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: receiving an image of a text to be rendered sent by a client; using a multimodal model to generate text based on the image of the text to be rendered, to obtain multiple key points and generated texts corresponding to the multiple key points, wherein the multiple key points are used to characterize the positions of the corresponding generated texts to be generated in the image of the text to be rendered; using the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points, to obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to characterize the area in the image of the text to be rendered where the target generated text is to be generated; and sending the target text area and the target generated text to the client.

[0008] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: in response to an input instruction acting on an operation interface, displaying an image of a text to be rendered on the operation interface; in response to a processing instruction acting on the operation interface, displaying a target text area and a target generated text on the operation interface, wherein the target text area is used to characterize an area in the image of the text to be rendered where the target generated text is to be generated, the target text area and the target generated text corresponding to the target text area are the results obtained by processing multiple key points and the generated texts corresponding to the multiple key points using a multimodal model, the multiple key points and the generated texts corresponding to the multiple key points are the results obtained by generating text based on the image of the text to be rendered using a multimodal model, and the multiple key points are used to characterize the position of the corresponding generated text to be generated in the image of the text to be rendered; in response to a generation instruction acting on the operation interface, displaying a target image on the operation interface, wherein the target image is an image obtained by fusing the target generated text with the image of the text to be rendered based on the target text area.

[0009] According to another aspect of an embodiment of the present application, an image generation method is also provided, including: obtaining an image of text to be rendered by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the image of the text to be rendered; using a multimodal model to generate text based on the image of the text to be rendered, to obtain multiple key points and generated texts corresponding to the multiple key points, wherein the multiple key points are used to characterize the positions of the corresponding generated texts to be generated in the image of the text to be rendered; using the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points, to obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to characterize the area in the image of the text to be rendered where the target generated text is to be generated; based on the target text area, fusing the target generated text with the image of the text to be rendered to obtain a target image; outputting the target image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0010] According to one aspect of an embodiment of the present application, an image processing device is provided, including: an acquisition module, used to acquire an image of a text to be rendered; a first processing module, used to generate text based on the image of the text to be rendered using a multimodal model, and obtain multiple key points and generated texts corresponding to the multiple key points, wherein the multiple key points are used to represent the positions in the image of the text to be rendered where the corresponding generated texts need to be rendered; a second processing module, used to process the multiple key points and the generated texts corresponding to the multiple key points using the multimodal model, and obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to represent the area in the image of the text to be rendered where the target generated text needs to be rendered; and a fusion module, used to fuse the target generated text with the image of the text to be rendered based on the target text area, and obtain a target image.

[0011] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: a capture module, used to capture a scene image in a real scene; a first processing module, used to generate text based on the scene image using a multimodal model, and obtain multiple key points and scene texts corresponding to the multiple key points, wherein the multiple key points are used to characterize the positions of the corresponding scene texts to be generated in the scene image; a second processing module, used to process the multiple key points and the scene texts corresponding to the multiple key points using the multimodal model, and obtain a target text area and a target scene text corresponding to the target text area, wherein the target text area is used to characterize the area in the scene image where the target scene text is to be generated; and a fusion module, used to fuse the target scene text with the scene image based on the target text area, and obtain a scene text image.

[0012] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: a receiving module, used to receive an image of a text to be rendered sent by a client; a first processing module, used to generate text based on the image of the text to be rendered using a multimodal model, and obtain multiple key points and generated texts corresponding to the multiple key points, wherein the multiple key points are used to characterize the positions of the corresponding generated texts to be generated in the image of the text to be rendered; a second processing module, used to process the multiple key points and the generated texts corresponding to the multiple key points using the multimodal model, and obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to characterize the area in the image of the text to be rendered where the target generated text is to be generated; and a sending module, used to send the target text area and the target generated text to the client.

[0013] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: a first display module, which is used to respond to an input instruction acting on an operation interface and display an image of a text to be rendered on the operation interface; a second display module, which is used to respond to a processing instruction acting on the operation interface and display a target text area and a target generated text on the operation interface, wherein the target text area is used to characterize an area in the image of the text to be rendered where the target generated text is to be generated, and the target text area and the target generated text corresponding to the target text area are the results obtained by processing multiple key points and the generated texts corresponding to the multiple key points using a multimodal model, and the multiple key points and the generated texts corresponding to the multiple key points are the results obtained by generating text based on the image of the text to be rendered using a multimodal model, and the multiple key points are used to characterize the position of the corresponding generated text to be generated in the image of the text to be rendered; a third display module, which is used to respond to a generation instruction acting on the operation interface and display a target image on the operation interface, wherein the target image is an image obtained by fusing the target generated text with the image of the text to be rendered based on the target text area.

[0014] According to another aspect of an embodiment of the present application, an image generating device is also provided, including: an acquisition module, used to acquire an image of a text to be rendered by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter includes the image of the text to be rendered; a first processing module, used to generate text based on the image of the text to be rendered using a multimodal model, and obtain multiple key points and generated texts corresponding to the multiple key points, wherein the multiple key points are used to characterize the positions of the corresponding generated texts to be generated in the image of the text to be rendered; a second processing module, used to process the multiple key points and the generated texts corresponding to the multiple key points using a multimodal model, and obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to characterize the area in the image of the text to be rendered where the target generated text is to be generated; a fusion module, used to fuse the target generated text with the image of the text to be rendered based on the target text area, and obtain a target image; an output module, used to output the target image by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter includes the target image.

[0015] According to another aspect of the embodiments of the present application, an electronic device is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method of each embodiment of the present application is executed when the program is running.

[0016] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0017] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0018] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0019] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0020] In an embodiment of the present application, an image of a text to be rendered is obtained; a multimodal model is used to generate text based on the image of the text to be rendered to obtain a plurality of key points and generated texts corresponding to the plurality of key points; a multimodal model is used to process the plurality of key points and the generated texts corresponding to the plurality of key points to obtain a target text area and a target generated text corresponding to the target text area; and a target generated text is fused with the image of the text to be rendered based on the target text area to obtain a target image. It is easy to notice that the multimodal model can be used to determine the key points and generate text through a multi-stage processing process, and then determine the reasonable target text area and target generated text. By decomposing the complex rendering task into easier, logic-based progressive tasks, the multimodal model can process the progressive tasks based on the thinking chain, thereby improving the processing accuracy of the multimodal model in the rendering task processing process, and achieving the purpose of accurately rendering text on the image and obtaining a generated image with minimal difference from the original image. Furthermore, based on the generated image with a small difference from the original image, it can ensure that the processing effect of the subsequent processing of the original image is better, thereby achieving the technical effect of improving the processing effect of the subsequent processing of the original image, and then solving the technical problem in the related technology that the generated image after the text is generated is large in difference from the original image, resulting in the deterioration of the processing effect of the subsequent processing of the original image based on the generated image.

[0021] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0023] Figure 1 is a schematic diagram of an application scenario of an image processing method according to an embodiment of the present application;

[0024] Figure 2 is a flowchart of an image processing method according to an embodiment of the present application;

[0025] Figure 3 is a system flow chart of an optional image processing method according to an embodiment of the present application;

[0026] Figure 4 is an optional flowchart of obtaining a target text area and a target generated text using a multimodal model according to an embodiment of the present application;

[0027] Figure 5is a flowchart of an image processing method according to an embodiment of the present application;

[0028] Figure 6 is a flowchart of an image processing method according to an embodiment of the present application;

[0029] Figure 7 is a flowchart of an image processing method according to an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of an optional operation interface according to an embodiment of the present application;

[0031] Fig. 9 is a flow chart of an image generating method according to an embodiment of the present application;

[0032] Fig.10 is a schematic diagram of an optional image processing device according to an embodiment of the present application;

[0033] Fig.11 is a schematic diagram of an optional image processing device according to an embodiment of the present application;

[0034] Fig.12 is a schematic diagram of an optional image processing device according to an embodiment of the present application;

[0035] Fig.13 is a schematic diagram of an optional image processing device according to an embodiment of the present application;

[0036] Fig.14 is a schematic diagram of an optional image generating device according to an embodiment of the present application;

[0037] Fig.15 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than 10 trillion model parameters. The large model can also be called a cornerstone model / foundation model (Foundation Model). The large model is pre-trained through large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as large-scale language model (Large Language Model, LLM), multi-modal pre-training model (multi-modal pre-training model), etc.

[0041] It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples, so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), image generation, etc. It can also be widely used in natural language processing tasks such as text-based sentiment classification, text summary generation, machine translation, etc. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, data processing through a multimodal model in an image processing scenario is taken as an example for explanation,

[0042] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0043] Scene text: text in natural images.

[0044] Reasonableness: The generated text area and content should be consistent with the context of the image and should not be meaningless.

[0045] Hint: Text input for large language models.

[0046] Bezier curve box: The area enclosed by the coordinates of the Bezier control points, connected by two curve segments and two line segments, where the curve segments can be converted into curve expressions through Bernstein polynomials.

[0047] Multimodal Large Language Model: A model that uses the text understanding ability of the large language model and combines it with visual perception. The visual input is usually encoded through a visual encoder and then processed into a form that the large language model can understand before inputting it into the large language model.

[0048] Bernstein polynomials: can be used to approximate complex curves and surfaces. By increasing the order of the polynomial, the accuracy of the approximation can be improved.

[0049] According to an embodiment of the present application, an image processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0050] Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the above-mentioned image processing method provided in the embodiment of the present application can be applied to the following Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of an image processing method according to an embodiment of the present application. Figure 1 In the application scenario shown, the large model is deployed in the server 10, and the server 10 can be connected to one or more client devices 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client device 20 here may include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc. The client device 20 can interact with the user through a graphical user interface to implement the call of the large model, thereby implementing the method provided in the embodiment of the present application.

[0051] In an embodiment of the present application, a system composed of a client device and a server can perform the following steps: the client device executes: displaying a target image in a graphical user interface, the server executes: obtaining an image of the text to be rendered, processing the image of the text to be rendered using a multimodal model to obtain multiple key points and generated texts corresponding to the key points, processing multiple key points and generated texts using a multimodal model to obtain a target text area and a target generated text, and merging the target generated text with the image of the text to be rendered based on the target text area to obtain a target image. It should be noted that, if the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiment of the present application can be performed in the client device.

[0052] Under the above operating environment, this application provides Figure 2 The image processing method shown. Figure 2 FIG. 1 is a flow chart of an image processing method according to an embodiment of the present application. Figure 2 As shown, the method may include the following steps:

[0053] Step S202, obtaining an image of the text to be rendered.

[0054] The above-mentioned image to be rendered with text may be an image that needs to be rendered with text, that is, an image that needs to add corresponding text describing real things into the image. The image contains at least one real thing in the real world, and the real thing may include but is not limited to: buildings, vehicles, natural scenery, animals, plants and humans.

[0055] In an optional embodiment, when it is necessary to accurately render text on an image, an image of the text to be rendered can be first obtained. For example, after the client captures a real scene to obtain an image containing "sky, grass and mountains" (i.e., the image of the text to be rendered), the client uploads the captured image of the text to be rendered to the server, and then the server can obtain the image of the text to be rendered. However, it is not limited to this, and it can also be that after the client captures a real scene to obtain an image containing "buildings, cars and humans", the client sends the captured image of the text to be rendered to the server, and then the server can obtain the image of the text to be rendered.

[0056] For another example, the server may directly obtain an image containing "sky, airplane, birds" (i.e., an image of text to be rendered) from a specific server. Alternatively, the server may directly obtain an image containing "sky, woods, and animals" from a specific server.

[0057] For another example, the client may send a video containing real things in the real world to the server, and the server may take a screenshot of the video to obtain an image of the text to be rendered, but the present invention is not limited thereto.

[0058] Step S204, using the multimodal model to generate text based on the image of the text to be rendered, to obtain multiple key points and generated text corresponding to the multiple key points, wherein the multiple key points are used to represent the positions in the image of the text to be rendered where the generated text needs to be rendered.

[0059] The above-mentioned multimodal model can be a model that integrates multiple different types of data and information sources, which can better understand the problem and provide more accurate and comprehensive solutions. The application fields of the multimodal model may include but are not limited to: natural language processing, computer vision, speech recognition, and machine learning. The above-mentioned key points can be optional positions in the image of the text to be rendered for generating text, wherein the optional positions can be blank areas in the image to be rendered, and can also be areas where real objects in the image of the text to be rendered are located, but are not limited to this. The above-mentioned generated text can be generated by the multimodal model in combination with the context of the image to be rendered after understanding, and the generated text can accurately describe the specific real objects in the image of the text to be rendered. For example, when the real objects in the image of the text to be rendered are "sky, buildings and vehicles", the content of the generated text can be: blue sky, buildings with glass structures on the outside, and cars.

[0060] In an optional embodiment, when an image of the text to be rendered is obtained, the multimodal model can first be used to generate text based on the image of the text to be rendered, and multiple key points and generated text corresponding to the multiple key points can be obtained. For example, after the image of the text to be rendered is obtained, the first prompt information of the image of the text to be rendered can be obtained first, and then the multimodal model can be used to process the image of the text to be rendered based on the first prompt information to obtain multiple key points and generated text corresponding to the multiple key points. Among them, the first prompt information is used to illustrate the processing tasks of the multimodal model, the attribute information of the key points, the output format of the key points, and the output format of the generated text corresponding to the key points.

[0061] Among them, the processing task can be: identifying the area in the image of the text to be rendered where text can be rendered, and identifying the content of specific things in the image of the text to be rendered, but not limited to this. The attribute information of the key point may include but is not limited to: the number, position (i.e., pixel coordinates), color, grayscale value, transparency, brightness, contrast, saturation, hue, texture, edge information, etc. of the key point in the image of the text to be rendered. The output format of the key point may include but is not limited to: RGB (Red Green Blue), RGBA (Red Green Blue Alpha), CMYK (Cyan Magenta Yellow Key), HSV (Hue, Saturation Value, hue, saturation, brightness), YCbCr (Luma Chroma Blue Chroma Red, brightness, blue color difference, red color difference). The output format of the generated text may include but is not limited to: font color, font size, font arrangement layout.

[0062] For example, based on the first prompt information, the attribute information may include quantity, position, color and brightness, the output format may be RGB key points, and the output format may include font color, font size and font arrangement layout of the generated text.

[0063] For another example, first, the image to be rendered with text can be preprocessed through a multimodal model (e.g., adjusting image size, brightness, contrast, and removing noise, etc.), then objects in the image can be detected based on a deep learning algorithm, and the location of the objects can be determined, and then the detected objects and the space around the objects can be analyzed. Usually, blank areas or background areas are ideal for adding text, and then image segmentation algorithms or deep learning algorithms can be used to further identify blank areas in the image, so as to find areas where text can be added and obtain key points, and finally the generated text corresponding to the objects and key points can be obtained from the preset database.

[0064] Step S206, using a multimodal model to process multiple key points and generated texts corresponding to the multiple key points, to obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to represent the area in the image of the text to be rendered where the target generated text needs to be rendered.

[0065] The above-mentioned target generated text may be the generated text that is processed and finally rendered in the image of the text to be rendered. In other words, the target generated text meets the rendering requirements of the user, where the rendering requirements may include but are not limited to: the content of the target generated text must accurately reflect the content of the image of the text to be rendered, and the rendering position of the target generated text must be reasonable. The above-mentioned target text area may be the area used to render the target generated text.

[0066] In an optional embodiment, when the multimodal model obtains the key points of the image of the text to be rendered and the generated text corresponding to the key points, the multimodal model can also process the key points and the generated text corresponding to the key points to obtain the target text area and the target generated text. For example, first, the second prompt information can be obtained, and then the multimodal model can be used to process multiple key points and the generated text corresponding to the key points based on the second prompt information to obtain the target text area and the target generated text. Among them, the second prompt information is used to characterize the processing task of the multimodal model, the representation form of the target text area, the output format of the target text area, and the output format of the target generated text.

[0067] The processing tasks may include but are not limited to: processing the generated text to obtain the target generated text, and processing the key points to obtain the target text area. The representation form of the target text area may include but are not limited to: a curved frame, a rectangular frame, a circular frame, and an irregular closed frame. The output format of the target text area may include but are not limited to: fully transparent, semi-transparent or opaque. The output format of the target generated text area may include but are not limited to: text row display, text column display, and text diagonal display. For example, based on the second prompt information, a target generated text area with a curved frame as the representation form and a semi-transparent output format, and a target generated text with a text row display as the output format can be obtained.

[0068] For another example, the key points and the generated text corresponding to the key points can also be input into a deep learning model, which will process the key points and the generated text based on the color, content layout, and specific objects of the image of the text to be rendered to obtain the target text area and target generated text that meet the user's rendering requirements.

[0069] Step S208 , based on the target text area, the target generated text is merged with the image of the text to be rendered to obtain a target image.

[0070] In an optional embodiment, when the target text area of ​​the image of the text to be rendered and the target generated text are obtained, the target generated text and the image of the text to be rendered are fused based on the target text area to obtain the target image. For example, the target text area can be converted into a specific position corresponding to the specific text content in the target generated text, and then based on the specific position, the specific text content in the target generated text can be fused with the image of the text to be rendered to obtain the target image. For another example, the specific content of the target generated text can be fused with the image of the text to be rendered directly based on the target text area to obtain the target image.

[0071] In an embodiment of the present application, an image of a text to be rendered is obtained; a multimodal model is used to generate text based on the image of the text to be rendered to obtain a plurality of key points and generated texts corresponding to the plurality of key points; a multimodal model is used to process the plurality of key points and the generated texts corresponding to the plurality of key points to obtain a target text area and a target generated text corresponding to the target text area; and a target generated text is fused with the image of the text to be rendered based on the target text area to obtain a target image. It is easy to notice that the multimodal model can be used to determine the key points and generate text through a multi-stage processing process, and then determine the reasonable target text area and target generated text. By decomposing the complex rendering task into easier, logic-based progressive tasks, the multimodal model can process the progressive tasks based on the thinking chain, thereby improving the processing accuracy of the multimodal model in the rendering task processing process, and achieving the purpose of accurately rendering text on the image and obtaining a generated image with minimal difference from the original image. Furthermore, based on the generated image with a small difference from the original image, it can ensure that the processing effect of the subsequent processing of the original image is better, thereby achieving the technical effect of improving the processing effect of the subsequent processing of the original image, and then solving the technical problem in the related technology that the generated image after the text is generated is large in difference from the original image, resulting in the deterioration of the processing effect of the subsequent processing of the original image based on the generated image.

[0072] In the above embodiment of the present application, a multimodal model is used to generate text based on an image of the text to be rendered, and multiple key points and generated texts corresponding to the multiple key points are obtained, including: obtaining first prompt information, wherein the first prompt information is used to characterize the processing task of the multimodal model, the attribute information of the multiple key points, the output format of the multiple key points, and the output format of the generated texts corresponding to the multiple key points; using the multimodal model to generate text based on the first prompt information and the image of the text to be rendered, and multiple key points and generated texts corresponding to the multiple key points are obtained.

[0073] The above-mentioned first prompt information can be input by the user to assist the multimodal model in generating key points and information of generated text corresponding to the key points. Among them, the content of the first prompt information can be: given a background image (i.e., the image of the text to be rendered) where the text (i.e., the target generated text) will be written, plan the location of the text and visual text for the image. The planned location (i.e., the target text area) is represented by the coordinates of the points (i.e., the key points), which should be located in an area of ​​the image suitable for writing text. The size of the background image is 512*512. Therefore, none of the coordinate attributes should exceed 512. The planned text (i.e., the target generated text) should be interrelated and suitable to appear at the position represented by the corresponding point.

[0074] In an optional embodiment, firstly, the first prompt information can be obtained, wherein the first prompt information includes the processing task of the multimodal model, the attribute information of the key points, the output format of the key points, and the output format of the generated text corresponding to the key points. Secondly, the multimodal model can be used to generate text based on the first prompt information and the image of the text to be rendered, so as to obtain multiple key points and the generated text corresponding to the multiple key points. For example, the first prompt information can be feature extracted to obtain text features, and then the image of the text to be rendered can be feature extracted to obtain image features. Finally, the multimodal model can be used to obtain the key points and the generated text corresponding to the key points based on the text features and the image features.

[0075] It should be noted that the pre-trained multimodal large model has sufficient understanding ability to determine the key point locations suitable for writing text on the image to be rendered. The text content is also generated by the multimodal large model, which is generated based on the location of the key points and the semantics of the image. Considering the consistency of the generated text content and the rationality of the arrangement of the text area, this application outputs the generated text in the form of text lines.

[0076] In the above embodiment of the present application, a multimodal model is used to generate text based on the first prompt information and the image of the text to be rendered, and multiple key points and generated texts corresponding to the multiple key points are obtained, including: using a word segmenter to extract features of the first prompt information to obtain text features; using a visual encoder to extract features of the image of the text to be rendered to obtain image features; using a projection layer to align the image features with the text features to obtain aligned image features; inputting the aligned image features and text features into a large language model, and obtaining multiple key points output by the multimodal model and generated texts corresponding to the multiple key points.

[0077] The above-mentioned text segmenter can be any one or more text segmenters in the large language model that can obtain text tokens, for example, it can include but is not limited to: Byte Pair Encoding (BPE), Word Piece Encoding (WPE), and Unigram Encoding (UE). The above-mentioned visual encoder can be any one or more visual encoders in the prior art that can obtain image features, for example, it can include but is not limited to: lossless encoder, lossy encoder, video encoder, neural network encoder, region-based encoder, block-based encoder, and transformation-based encoder. The above-mentioned projection layer (Projection layer) is used to align image features with the text embedding input adapted by the subsequent pre-trained large language model. The types of projection layers may include but are not limited to: fully connected layer, linear projection layer, nonlinear projection layer, cyclic projection layer and attention projection layer.

[0078] In an optional embodiment, when the first prompt information is obtained, the first prompt information can be first tokenized by a text segmenter to obtain a text token. At the same time, a visual encoder can be used to extract features of the image of the rendered text to obtain image features. Then, the image features and the text tokens can be aligned through a projection layer to obtain aligned image features and aligned text tokens. Finally, the aligned image features and aligned text tokens can be input into a large language model to obtain multiple key points output by the large language model and generated text corresponding to the key points.

[0079] In the above embodiment of the present application, a multimodal model is used to process multiple key points and generated texts corresponding to the multiple key points to obtain a target text area and a target generated text corresponding to the target text area, including: obtaining second prompt information, wherein the second prompt information is used to characterize the processing task of the multimodal model, the representation form of the target text area, the output format of the target text area, and the output format of the target generated text; and a multimodal model is used to process multiple key points and generated texts corresponding to the multiple key points based on the second prompt information to obtain a target text area and a target generated text.

[0080] The above-mentioned second prompt information can be input in advance by the user, and is used to assist in generating prompt information for the target text area and the target generated text. Among them, the second prompt information can be: according to the above-mentioned planned points and text, plan the layout of the visual text for the image (i.e., the target text area). The key point should be located at the layout position, and the planned layout should be close to the point. The layout is expressed in the form of Bezier curve control point coordinates, representing the area on the image suitable for writing visual text. Each area consists of eight vertices, starting from the upper left corner and arranged in counterclockwise order. Appropriate layouts should not overlap with each other. The aspect ratio of the layout box needs to take into account the number of characters in the text. The more characters, the larger the aspect ratio.

[0081] In an optional embodiment, when the generated text corresponding to the key points and key segments is obtained, a second prompt information can also be obtained, wherein the second prompt information includes the processing task of the multimodal model, the representation form of the target text area, the output format of the target text area, and the output format of the target generated text. Then, the multimodal model can be used to process multiple key points and the generated text corresponding to the multiple key points based on the second prompt information to obtain the target text area and the target generated text. For example, the multimodal model can be used to process multiple key points based on the second prompt information and the number of characters in the generated text corresponding to the multiple key points to obtain the target text area.

[0082] In the above embodiment of the present application, a multimodal model is used to process multiple key points and generated texts corresponding to the multiple key points based on the second prompt information to obtain a target text area, including: using a multimodal model to process multiple key points based on the second prompt information and the number of characters of the generated texts corresponding to the multiple key points to obtain a Bezier curve corresponding to the target text area; based on the positions of multiple control points of the Bezier curve, the target text area is obtained.

[0083] In an optional embodiment, a multimodal model can also be used to process multiple key points based on the second prompt information and the number of characters of the generated text corresponding to the key points to obtain the Bezier curve corresponding to the target text area, and then the target text area can be obtained based on the positions of multiple control points of the Bezier curve. For example, in order to better simulate the various text forms in real scenes, a curved bounding box can also be generated based on the Bezier curve. The coordinates of the curved text area are represented by eight Bezier control points, and the area surrounded by two curve segments and two line segments is the target text area.

[0084] It should be noted that the text area and content output in language format are strings, and the visual image output is a picture. The advantage of outputting in language format is that the information density is high. If the mask is output in visual form, there will be a lot of redundant information. In addition, the output in language format is convenient for subsequent modification and other operations, and the interaction is more friendly.

[0085] In the above embodiment of the present application, the target generated text is fused with the image of the text to be rendered based on the target text area to obtain the target image, including: converting the target text area into a target word position, wherein the target word position is used to characterize the position of the words in the target generated text in the image of the text to be rendered; based on the target word position, the words in the target generated text are fused with the image of the text to be rendered to obtain the target image.

[0086] In an optional embodiment, first, the target text area can be converted into a bounding box of the target word (i.e., the target word position) through post-processing, and then based on the target word position, the words in the target generated text can be fused with the image of the text to be rendered to obtain the target image.

[0087] In the above embodiment of the present application, the method also includes: outputting multiple key points and generated texts corresponding to the multiple key points; when receiving a first feedback result, using a multimodal model to process the first feedback result and the first unmodified result to obtain a target text area and a target generated text, wherein the first feedback result is a result obtained by modifying multiple key points and / or the generated texts corresponding to the multiple key points, and the first unmodified result is used to characterize the results other than the first feedback result in the multiple key points and the generated texts corresponding to the multiple key points.

[0088] In an optional embodiment, after the multimodal model outputs the key points and the generated text corresponding to the key points, the client can receive the key points and the generated text corresponding to the key points and display them on the display interface. At this time, the user can intuitively see whether the key points and the generated text corresponding to the key points meet the user's needs (that is, whether they are reasonable). When the key points or the generated text corresponding to the key points do not meet the user's needs, for example, if the position of the key points does not match the rendering position in the image of the text to be rendered preset by the user, or the content of the generated text does not match the content of the image of the text to be rendered, the user can modify the key points or the generated text corresponding to the key points to obtain a first feedback result, wherein the first feedback result includes the modified key points or the generated text corresponding to the modified key points, and then the client can return the first feedback result to the server. After receiving the first feedback result, the server can use the multimodal model to process the first feedback result and the first non-feedback result to obtain the target text area and the target generated text, wherein the first non-feedback result can be the result that the key points and the generated text are not modified.

[0089] In the above embodiment of the present application, the method also includes: adjusting the multimodal model based on the image of the text to be rendered and the first feedback result.

[0090] In an optional embodiment, after obtaining the first feedback result, the server may fine-tune the multimodal model based on the image of the rendered text and the first feedback result to improve the accuracy of the multimodal model.

[0091] In the above embodiment of the present application, the method also includes: outputting a target text area and a target generated text; when a second feedback result is received, fusing the modified generated text with the image of the text to be rendered based on the modified text area to obtain a target image, or fusing the target generated text with the image of the text to be rendered based on the modified text area to obtain a target image, or fusing the modified generated text with the image of the text to be rendered based on the target text area to obtain a target image, wherein the second feedback result is a result obtained by modifying the target text area and / or the target generated text.

[0092] In an optional embodiment, after the multimodal model outputs the target text area and the target generated text, the target text area and the target generated text can be sent to the client. When the client receives the target text area and the target generated text, it can display the target text area and the target generated text to the user on a display interface. At this time, if the target text area or the target generated text does not meet the user's needs, the user can modify the target text area or the target generated text to obtain a second feedback result, wherein the second feedback result includes the modified target text area or the target generated text. Then the client can return the second feedback result to the server. After receiving the second feedback result, the server can fuse the modified generated text with the image of the text to be rendered based on the modified text area to obtain the target image, or fuse the target generated text with the image of the text to be rendered based on the modified text area to obtain the target image, or fuse the modified generated text with the image of the text to be rendered based on the target text area to obtain the target image.

[0093] In the above embodiment of the present application, the method further includes: adjusting the multimodal model based on the image of the text to be rendered and the second feedback result.

[0094] In an optional embodiment, after obtaining the second feedback result, the server may fine-tune the multimodal model based on the image of the rendered text and the second feedback result to improve the accuracy of the multimodal model.

[0095] Figure 3 is a system flow chart of an optional image processing method according to an embodiment of the present application, wherein the visual encoder, the projection layer and the large language model together constitute a multimodal large model, such as Figure 3 As shown, firstly, a background image (i.e., an image of the text to be rendered) is obtained, and then, the visual encoder can be used to extract features of the background image to obtain image features. At the same time, a predefined prompt (i.e., prompt information) input by the user can be received, and the text segmenter can be used to extract features of the predefined prompt to obtain text features. Then, the image features and text features can be input into the projection layer to obtain aligned text features and aligned image features. Finally, the aligned text features and aligned image features can be input into the large language model, so that the text area (i.e., the target text area) and content (i.e., the target generated text) of the background image can be obtained.

[0096] Among them, the multimodal large model plays a vital role in various reasoning tasks such as reasoning detection and reasoning segmentation due to its powerful visual reasoning ability. Inspired by this, we try to fine-tune the multimodal large model and develop its ability to recommend text regions and content to achieve highly reasonable visual text generation. The scheme proposed in this application uses efficient fine-tuning technology to effectively fine-tune the multimodal large language model, and simultaneously turns on the training of the visual encoder and projection layer. The scheme takes predefined prompts and background images as input, and outputs text regions and content in language format. Compared with previous work, this application does not require additional feature maps (dense depth, semantic segmentation, 3D scene information, etc.) to generate text regions, making it more convenient to use.

[0097] Figure 4 is an optional flowchart of obtaining a target text area and a target generated text using a multimodal model according to Example 1 of the present application, such as Figure 4 As shown, first, the first prompt information can be input: plan the text content and point position of the visual text for the image, and the multimodal model can output the point coordinates and the text contents. Secondly, the second prompt information can be input: plan the region of the visual text for the image based on the point position and text content, and the multimodal model can output the target text region (region coordinate).

[0098] For predefined input prompts, the task can be decomposed into two steps, such as Figure 4As shown, the context understanding ability of the multimodal large language model is fully utilized. In the first step, the scheme finds reasonable key points on the image for text rendering and generates semantically reasonable text content for each location. In the second step, based on the key points and text content generated in the first step, the scheme identifies detailed text regions. There are two reasons for this design: (1) It decomposes a complex task into two easier progressive tasks, which is consistent with the thinking chain of the large language model when answering questions. (2) The generated text region can refer to the number of characters in the text content to better determine the region size. Considering the consistency of the generated text content and the rationality of the arrangement of the text region, the text content can be output in the form of text lines. The bounding box of the text line can be converted into the bounding box of each word through post-processing. In order to better simulate the various text forms in real scenes, a curved bounding box can also be generated based on the Bezier curve. The coordinates of the curved text region are represented by eight Bezier control points, surrounded by two curve segments and two line segments.

[0099] This scheme proposes a text region and content recommendation method based on a multimodal large model. It uses the powerful image understanding and visual reasoning capabilities of the multimodal large model to find reasonable text regions and determine reasonable text content on real background images, and then generates text through a general text generation method to achieve real scene text image generation.

[0100] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0101] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), including several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0103] According to an embodiment of the present application, an image processing method embodiment is also provided. Figure 5 is a flowchart of an image processing method according to an embodiment of the present application, such as Figure 5 As shown, the method comprises the following steps:

[0104] Step S502, capturing a scene image in a real scene;

[0105] Step S504, using the multimodal model to generate text based on the scene image, to obtain a plurality of key points and scene texts corresponding to the plurality of key points, wherein the plurality of key points are used to represent the positions of the corresponding scene texts to be generated in the scene image;

[0106] Step S506, using a multimodal model to process the multiple key points and the scene texts corresponding to the multiple key points to obtain a target text area and a target scene text corresponding to the target text area, wherein the target text area is used to represent the area in the scene image where the target scene text is to be generated;

[0107] Step S508: fusing the target scene text with the scene image based on the target text region to obtain a scene text image.

[0108] The above-mentioned real scene may be a scene in the real world containing a variety of real things, wherein the multiple real things may include but are not limited to: buildings, natural scenery, animals, plants and humans. The above-mentioned scene image may include at least one real thing. The above-mentioned scene text may be a text describing the image content of the scene image. The above-mentioned target text area may be an area in the scene image that is in the same position as the real thing to be described. The above-mentioned target scene text may be a text in the target text area that can accurately describe the real thing to be described.

[0109] In an optional embodiment, when it is necessary to accurately render text on an image of rendered text, first, a scene image in a real scene can be captured, and then a multimodal model can be used to generate text based on the scene image to obtain multiple key points and scene texts corresponding to the multiple key points. Then, the multimodal model can be used to process the multiple key points and the scene texts corresponding to the multiple key points to obtain a target text area and a target scene text corresponding to the target text area. Finally, the target scene text can be fused with the scene image based on the target text area to obtain a scene text image.

[0110] Among them, the multiple key points are used to represent the positions of the corresponding scene texts to be generated in the scene image, and the target text area is used to represent the area of ​​the target scene text to be generated in the scene image.

[0111] In the above embodiment of the present application, a multimodal model is used to generate text based on a scene image to obtain multiple key points and scene texts corresponding to the multiple key points, including: obtaining first prompt information, wherein the first prompt information is used to characterize the processing task of the multimodal model, the attribute information of the multiple key points, the output format of the multiple key points, and the output format of the scene text corresponding to the key points; and a multimodal model is used to generate text based on the first prompt information and the scene image to obtain multiple key points and scene texts corresponding to the multiple key points.

[0112] In an optional embodiment, firstly, the first prompt information can be obtained, and then the scene image can be processed by the multimodal model based on the first prompt information to obtain multiple key points and scene texts corresponding to the key points. The first prompt information is used to characterize the processing task of the multimodal model, the attribute information of the key points, the output format of the key points, and the output format of the scene text corresponding to the key points.

[0113] In the above embodiment of the present application, a multimodal model is used to process multiple key points and scene texts corresponding to the multiple key points to obtain a target text area and a target scene text corresponding to the target text area, including: obtaining second prompt information, wherein the second prompt information is used to characterize the processing task of the multimodal model, the representation form of the target text area, the output grid of the target text area, and the output format of the target scene text; and a multimodal model is used to process multiple key points and scene texts corresponding to the multiple key points based on the second prompt information to obtain a target text area and a target scene text.

[0114] In an optional embodiment, first, second prompt information may be obtained, and secondly, a multimodal model may be used to process multiple key points and scene texts corresponding to the multiple key points based on the second prompt information to obtain a target text region and a target scene text. The second prompt information is used to characterize the processing task of the multimodal model, the representation form of the target text region, and the output format of the target text region and the target scene text.

[0115] According to an embodiment of the present application, an image processing method embodiment is also provided. Figure 6 is a flowchart of an image processing method according to an embodiment of the present application, such as Figure 6 As shown, the method comprises the following steps:

[0116] Step S602, receiving an image of text to be rendered sent by the client;

[0117] Step S604, using the multimodal model to generate text based on the image of the text to be rendered, to obtain a plurality of key points and generated texts corresponding to the plurality of key points, wherein the plurality of key points are used to represent the positions of the corresponding generated texts to be generated in the image of the text to be rendered;

[0118] Step S606, using a multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points, to obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to represent the area where the target generated text is to be generated in the image of the text to be rendered;

[0119] Step S608, sending the target text area and the target generated text to the client.

[0120] In an optional embodiment, when it is necessary to accurately render text on an image of text to be rendered, first, the image of the text to be rendered sent by the client can be received, and then the multimodal model can be used to generate text based on the image of the text to be rendered to obtain multiple key points and generated texts corresponding to the multiple key points, and then the multimodal model can be used to process the multiple key points and the generated texts corresponding to the multiple key points to obtain the target text area and the target generated text, and finally the target text area and the target generated text can be sent to the client. Among them, the multiple key points are used to characterize the position of the corresponding generated text to be generated in the image of the text to be rendered, and the target text area is used to characterize the area of ​​the target generated text to be generated in the image of the text to be rendered.

[0121] According to an embodiment of the present application, an image processing method embodiment is also provided. Figure 7 is a flowchart of an image processing method according to an embodiment of the present application, such as Figure 7 As shown, the method comprises the following steps:

[0122] Step S702, in response to an input instruction acting on the operation interface, displaying an image of the text to be rendered on the operation interface;

[0123] Step S704, in response to the processing instruction acting on the operation interface, displaying the target text area and the target generated text corresponding to the target text area on the operation interface, wherein the target text area is used to represent the area in the image of the text to be rendered where the target generated text is to be generated, the target text area and the target generated text are the results obtained by processing a plurality of key points and the generated texts corresponding to the plurality of key points using a multimodal model, the plurality of key points and the generated texts corresponding to the plurality of key points are the results obtained by performing text generation based on the image of the text to be rendered using the multimodal model, and the plurality of key points are used to represent the position of the corresponding generated text to be generated in the image of the text to be rendered;

[0124] Step S706 , in response to the generation instruction acting on the operation interface, a target image is displayed on the operation interface, wherein the target image is an image obtained by fusing the target generated text with the image of the text to be rendered based on the target text area.

[0125] In an optional embodiment, Figure 8 is a schematic diagram of an optional operation interface according to an embodiment of the present application, such as Figure 8 As shown, the operation interface includes: an input button, a processing button, a generation button and a display area. When it is necessary to accurately render the text of the image to be rendered, the user can first perform an operation on the input button in the operation interface (for example, a click operation, a press operation or a slide operation), and an input instruction can be generated based on the operation. The operation interface responds to the input instruction and can display the image of the text to be rendered in the display area. Secondly, the user can perform an operation on the processing button (for example, a click operation, a press operation or a slide operation), and a processing instruction can be generated based on the operation. At this time, the operation interface can display the target text area and the target generated text in the display area of ​​the operation interface in response to the processing instruction. Finally, the user can perform an operation on the generation button (for example, a click operation, a press operation or a slide operation), and a generation instruction can be generated based on the operation. The operation interface responds to the generation instruction and can display the target image in the display area of ​​the operation interface.

[0126] Among them, the target text area is used to represent the area of ​​the target generated text to be generated in the image of the text to be rendered, the target text area and the target generated text are the results of processing multiple key points and the generated texts corresponding to the multiple key points using a multimodal model, the multiple key points and the generated texts corresponding to the multiple key points are the results of processing the image of the text to be rendered using a multimodal model, the multiple key points are used to represent the position of the corresponding generated text to be generated in the image of the text to be rendered, and the target image is the image obtained by fusing the target generated text with the image of the text to be rendered based on the target text area.

[0127] According to an embodiment of the present application, an image generating method embodiment is also provided. Fig. 9 is a flow chart of an image generation method according to an embodiment of the present application, such as Fig. 9 As shown, the method comprises the following steps:

[0128] Step S902, acquiring an image of the text to be rendered by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes the image of the text to be rendered;

[0129] Step S904, using the multimodal model to generate text based on the image of the text to be rendered, to obtain a plurality of key points and generated texts corresponding to the plurality of key points, wherein the plurality of key points are used to represent the positions of the corresponding generated texts to be generated in the image of the text to be rendered;

[0130] Step S906, using a multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points, to obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to represent the area in the image of the text to be rendered where the target generated text is to be generated;

[0131] Step S908, merging the target generated text with the image of the text to be rendered based on the target text area to obtain a target image;

[0132] Step S9010: outputting a target image by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target image.

[0133] The first interface mentioned above may be an interface for acquiring an image to be rendered from a client, and the second interface mentioned above may be an interface for outputting a target image to the client.

[0134] In an optional embodiment, when it is necessary to accurately render text on an image of text to be rendered, first, the image of the text to be rendered can be obtained from the client through the first interface, and secondly, the multimodal model can be used to generate text based on the image of the text to be rendered to obtain generated text corresponding to multiple key points and key points of the target text area, and then the multimodal model can be used to process the generated text corresponding to the multiple key points and key points of the target text area to obtain the target text area and the target generated text, and then the target generated text can be fused with the image of the text to be rendered based on the target text area to obtain the target image, and finally the target image can be output to the client through the second interface.

[0135] Among them, the first interface includes a first parameter, the parameter value of the first parameter includes an image of the text to be rendered; the target text area key point is used to represent the position of the corresponding generated text to be generated in the image of the text to be rendered; the target text area is used to represent the area of ​​the target generated text to be generated in the image of the text to be rendered; the second interface includes a second parameter, the parameter value of the second parameter includes the target image.

[0136] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Fig.10 is a schematic diagram of an optional image processing device according to an embodiment of the present application, such as Fig.10 As shown, the device includes: an acquisition module 1002 , a first processing module 1004 , a second processing module 1006 and a fusion module 1008 .

[0137] Among them, the acquisition module is used to acquire the image of the text to be rendered; the first processing module is used to use the multimodal model to generate text based on the image of the text to be rendered, and obtain multiple key points and generated texts corresponding to the multiple key points; the second processing module is used to use the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points, and obtain the target text area and the target generated text corresponding to the target text area; the fusion module is used to fuse the target generated text with the image of the text to be rendered based on the target text area to obtain the target image.

[0138] It should be noted that the acquisition module 1002, the first processing module 1004, the second processing module 1006 and the fusion module 1008 correspond to steps S202 to S208 in Example 1, and the four modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and may run in the server 10 provided in Example 1.

[0139] In the above embodiment of the present application, the first processing module includes: a first acquisition unit and a first processing unit.

[0140] Among them, the first acquisition unit is used to obtain the first prompt information; the first processing unit is used to use the multimodal model to generate text based on the first prompt information and the image of the text to be rendered, and obtain multiple key points and generated texts corresponding to the multiple key points.

[0141] In the above embodiment of the present application, the processing unit includes: a first extraction subunit, a second extraction subunit, an alignment subunit and an input subunit.

[0142] Among them, the first extraction subunit is used to use the text segmenter to perform feature extraction on the first prompt information to obtain text features; the second extraction subunit is used to use the visual encoder to perform feature extraction on the image of the rendered text to obtain image features; the alignment subunit is used to use the projection layer to align the image features with the text features to obtain aligned image features and aligned text features; the input subunit is used to input the aligned image features and the aligned text features into the multimodal model, and obtain multiple key points output by the multimodal model and the generated texts corresponding to the multiple key points.

[0143] In the above embodiment of the present application, the second processing module includes: a second acquisition unit and a second processing unit.

[0144] Among them, the second acquisition unit is used to obtain the second prompt information; the second processing unit is used to use the multimodal model to process multiple key points and generated texts corresponding to the multiple key points based on the second prompt information to obtain the target text area and the target generated text.

[0145] In the above embodiment of the present application, the second processing unit includes: a first processing sub-unit and a second processing sub-unit.

[0146] Among them, the first processing subunit is used to use a multimodal model to process multiple key points based on the second prompt information and the number of characters of the generated text corresponding to the multiple key points to obtain the Bezier curve corresponding to the target text area; the second processing subunit is used to obtain the target text area based on the positions of multiple control points of the Bezier curve.

[0147] In the above embodiments of the present application, the fusion module includes: a conversion unit and a fusion unit.

[0148] The conversion unit is used to convert the target text area into the target word position; the fusion unit is used to fuse the words in the target generated text with the image of the text to be rendered based on the target word position to obtain the target image.

[0149] In the above embodiments of the present application, the device further includes: a first output module and a processing module.

[0150] Among them, the first output module is used to output multiple key points and generated texts corresponding to the multiple key points; the processing module is used to use a multimodal model to process the first feedback result and the first unmodified result when receiving the first feedback result, so as to obtain the target text area and the target generated text.

[0151] In the above embodiments of the present application, the device further includes: a first adjustment module.

[0152] The first adjustment module is used to adjust the multimodal model based on the image of the text to be rendered and the first feedback result.

[0153] In the above embodiments of the present application, the device further includes: a second output module and a text and image fusion module.

[0154] Among them, the second output module is used to output the target text area and the target generated text; the text and image fusion module is used to, when receiving the second feedback result, fuse the modified generated text with the image of the text to be rendered based on the modified text area to obtain the target image, or fuse the target generated text with the image of the text to be rendered based on the modified text area to obtain the target image, or fuse the modified generated text with the image of the text to be rendered based on the target text area to obtain the target image.

[0155] In the above embodiments of the present application, the device further includes: a second adjustment module.

[0156] The second adjustment module is used to adjust the multimodal model based on the image of the text to be rendered and the second feedback result.

[0157] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0158] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Fig.11 is a schematic diagram of an optional image processing device according to an embodiment of the present application, such as Fig.11 As shown, the device includes: a capture module 1102 , a first processing module 1104 , a second processing module 1106 and a fusion module 1108 .

[0159] Among them, the capture module is used to capture scene images in real scenes; the first processing module is used to use a multimodal model to generate text based on the scene image, and obtain multiple key points and scene texts corresponding to the multiple key points; the second processing module is used to use the multimodal model to process multiple key points and scene texts corresponding to the multiple key points, and obtain a target text area and a target scene text corresponding to the target text area; the fusion module is used to fuse the target scene text with the scene image based on the target text area to obtain a scene text image.

[0160] It should be noted that the above-mentioned capture module 1102, first processing module 1104, second processing module 1106 and fusion module 1108 correspond to steps S502 to S508 in Example 2, and the four modules and corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and may be run in the server 10 provided in Example 1.

[0161] In the above embodiment of the present application, the first processing module includes: a first acquisition unit and a first processing unit.

[0162] Among them, the first acquisition unit is used to obtain the first prompt information; the first processing unit is used to use the multimodal model to generate text based on the first prompt information and the scene image to obtain multiple key points and scene texts corresponding to the multiple key points.

[0163] In the above embodiment of the present application, the second processing module includes: a second acquisition unit and a second processing unit.

[0164] Among them, the second acquisition unit is used to obtain the second prompt information; the second processing unit is used to use the multimodal model to process multiple key points and scene texts corresponding to the multiple key points based on the second prompt information to obtain the target text area and the target scene text.

[0165] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0166] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Fig.12 is a schematic diagram of an optional image processing device according to an embodiment of the present application, such as Fig.12 As shown, the device includes: a receiving module 1202 , a first processing module 1204 , a second processing module 1206 and a sending module 1208 .

[0167] Among them, the receiving module is used to receive an image of the text to be rendered sent by the client; the first processing module is used to use the multimodal model to generate text based on the image of the text to be rendered, and obtain multiple key points and generated texts corresponding to the multiple key points; the second processing module is used to use the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points, and obtain the target text area and the target generated text corresponding to the target text area; the sending module is used to send the target text area and the target generated text to the client.

[0168] It should be noted that the receiving module 1202, the first processing module 1204, the second processing module 1206 and the sending module 1208 correspond to steps S602 to S608 in Example 3, and the four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and may run in the server 10 provided in Example 1.

[0169] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0170] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image processing method is also provided. Fig.13 is a schematic diagram of an optional image processing device according to an embodiment of the present application, such as Fig.13 As shown, the device includes: a first display module 1302 , a second display module 1304 and a third display module 1306 .

[0171] Among them, the first display module is used to respond to the input instructions acting on the operation interface, and display the image of the text to be rendered on the operation interface; the second display module is used to respond to the processing instructions acting on the operation interface, and display the target text area and the target generated text corresponding to the target text area on the operation interface; the third display module is used to respond to the generation instructions acting on the operation interface, and display the target image on the operation interface.

[0172] It should be noted that the first display module 1302, the second display module 1304 and the third display module 1306 correspond to steps S702 to S706 in Example 4, and the three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in Example 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be part of a device and run in the server 10 provided in Example 1.

[0173] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0174] According to an embodiment of the present application, an image processing device for implementing the above-mentioned image generation method is also provided. Fig.14 is a schematic diagram of an optional image generating device according to an embodiment of the present application, such as Fig.14 As shown, the device includes: an acquisition module 1402 , a first processing module 1404 , a second processing module 1406 , a fusion module 1408 and an output module 14010 .

[0175] Among them, the acquisition module is used to obtain the image of the text to be rendered by calling the first interface; the first processing module is used to use the multimodal model to generate text based on the image of the text to be rendered, and obtain multiple key points and generated texts corresponding to the multiple key points; the second processing module is used to use the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points, and obtain the target text area and the target generated text corresponding to the target text area; the fusion module is used to fuse the target generated text with the image of the text to be rendered based on the target text area to obtain the target image; the output module is used to output the target image by calling the second interface.

[0176] It should be noted that the acquisition module 1402, the first processing module 1404, the second processing module 1406, the fusion module 1408 and the output module 14010 correspond to steps S902 to S9010 in Example 5, and the five modules and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above-mentioned modules may also be part of the device and may be run in the server 10 provided in Example 1.

[0177] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0178] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0179] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0180] In this embodiment, the electronic device can execute the program code in the method.

[0181] Optionally, Fig.15 is a structural block diagram of an electronic device according to an embodiment of the present application. Fig.15 As shown, the electronic device A may include: one or more (only one is shown in the figure) processors 1502, a memory 1504, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0182] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0183] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain an image of the text to be rendered; use a multimodal model to generate text based on the image of the text to be rendered to obtain multiple key points and generated texts corresponding to the multiple key points; use the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points to obtain a target text area and a target generated text corresponding to the target text area; based on the target text area, fuse the target generated text with the image of the text to be rendered to obtain a target image.

[0184] Optionally, the processor may also execute the program code of the following steps: obtaining first prompt information; generating text based on the first prompt information and the image of the text to be rendered using a multimodal model to obtain multiple key points and generated text corresponding to the multiple key points.

[0185] Optionally, the processor may also execute the program code of the following steps: using a text segmenter to perform feature extraction on the first prompt information to obtain text features; using a visual encoder to perform feature extraction on the image of the rendered text to obtain image features; using a projection layer to align the image features with the text features to obtain aligned image features and aligned text features; inputting the aligned image features and the aligned text features into a multimodal model, and obtaining multiple key points output by the multimodal model and generated texts corresponding to the multiple key points.

[0186] Optionally, the processor may also execute the program code of the following steps: obtaining second prompt information; processing multiple key points and generated texts corresponding to the multiple key points based on the second prompt information using a multimodal model to obtain a target text area and a target generated text.

[0187] Optionally, the processor may also execute the program code of the following steps: using a multimodal model to process multiple key points based on the second prompt information and the number of characters of the generated text corresponding to the multiple key points to obtain a Bezier curve corresponding to the target text area; and obtaining the target text area based on the positions of multiple control points of the Bezier curve.

[0188] Optionally, the processor may also execute program code of the following steps: converting the target text area into a target word position; based on the target word position, fusing the words in the target generated text with the image of the text to be rendered to obtain a target image.

[0189] Optionally, the processor may also execute the program code of the following steps: outputting multiple key points and generated texts corresponding to the key points; upon receiving a first feedback result, processing the first feedback result and the first unmodified result using a multimodal model to obtain a target text area and a target generated text.

[0190] Optionally, the processor may also execute program code of the following steps: adjusting the multimodal model based on the image of the text to be rendered and the first feedback result.

[0191] Optionally, the processor may also execute the program code of the following steps: outputting a target text area and a target generated text; upon receiving a second feedback result, fusing the modified generated text with the image of the text to be rendered based on the modified text area to obtain a target image, or fusing the target generated text with the image of the text to be rendered based on the modified text area to obtain a target image, or fusing the modified generated text with the image of the text to be rendered based on the target text area to obtain a target image.

[0192] Optionally, the processor may also execute program code of the following steps: adjusting the multimodal model based on the image of the text to be rendered and the second feedback result.

[0193] An embodiment of the present application provides a method for obtaining an image of text to be rendered; using a multimodal model to generate text based on the image of the text to be rendered to obtain multiple key points and generated texts corresponding to the multiple key points; using the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points to obtain a target text area and a target generated text corresponding to the target text area; based on the target text area, the target generated text is merged with the image of the text to be rendered to obtain a target image. It is easy to notice that the multimodal model can be used to determine the key points and generate text through a multi-stage processing process, and then determine the reasonable target text area and target generated text. By decomposing the complex rendering task into easier, logic-based progressive tasks, the multimodal model can process the progressive tasks based on the thinking chain, thereby improving the processing accuracy of the multimodal model in the rendering task processing process, and achieving the purpose of accurately rendering text on the image and obtaining a generated image with minimal difference from the original image. Furthermore, based on the generated image with a small difference from the original image, it can ensure that the processing effect of the subsequent processing of the original image is better, thereby achieving the technical effect of improving the processing effect of the subsequent processing of the original image, and then solving the technical problem in the related technology that the generated image after the text is generated is large in difference from the original image, resulting in the deterioration of the processing effect of the subsequent processing of the original image based on the generated image.

[0194] It will be understood by those skilled in the art that the structure shown in the figure is for illustration only, and the electronic device may also be a terminal device such as a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, and a mobile Internet device (Mobile Internet Devices, MID), PAD, etc. The figure does not limit the structure of the above-mentioned electronic device. For example, the electronic device A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in the figure, or have a configuration different from that shown in the figure.

[0195] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0196] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0197] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0198] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining an image of the text to be rendered; using a multimodal model to generate text based on the image of the text to be rendered, to obtain multiple key points and generated texts corresponding to the multiple key points; using the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points, to obtain a target text area and a target generated text corresponding to the target text area; based on the target text area, the target generated text is merged with the image of the text to be rendered to obtain a target image.

[0199] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: obtaining first prompt information; using a multimodal model to generate text based on the image of the text to be rendered based on the first prompt information, to obtain multiple key points and generated text corresponding to the multiple key points.

[0200] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: using a text segmenter to extract features from the first prompt information to obtain text features; using a visual encoder to extract features from the image of the rendered text to obtain image features; using a projection layer to align the image features with the text features to obtain aligned image features and aligned text features; inputting the aligned image features and the aligned text features into a multimodal model, and obtaining multiple key points output by the multimodal model and generated texts corresponding to the multiple key points.

[0201] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: obtaining second prompt information; using a multimodal model to process multiple key points and generated texts corresponding to the multiple key points based on the second prompt information to obtain a target text area and a target generated text.

[0202] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: using a multimodal model to process multiple key points based on the second prompt information and the number of characters of the generated text corresponding to the multiple key points to obtain a Bezier curve corresponding to the target text area; and obtaining the target text area based on the positions of multiple control points of the Bezier curve.

[0203] Optionally, the computer-readable storage medium is also configured to store program code for executing the following steps: converting a target text area into a target word position; based on the target word position, fusing the words in the target generated text with the image of the text to be rendered to obtain a target image.

[0204] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: outputting multiple key points and generated texts corresponding to the key points; upon receiving a first feedback result, processing the first feedback result and the first unmodified result using a multimodal model to obtain a target text area and a target generated text.

[0205] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: adjusting the multimodal model based on the image of the text to be rendered and the first feedback result.

[0206] Optionally, the computer-readable storage medium is also configured to store program code for performing the following steps: outputting a target text area and a target generated text; upon receiving a second feedback result, fusing the modified generated text with an image of the text to be rendered based on the modified text area to obtain a target image, or fusing the target generated text with an image of the text to be rendered based on the modified text area to obtain a target image, or fusing the modified generated text with an image of the text to be rendered based on the target text area to obtain a target image.

[0207] Optionally, the computer-readable storage medium is further configured to store program code for performing the following steps: adjusting the multimodal model based on the image of the text to be rendered and the second feedback result.

[0208] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0209] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0210] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0211] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0212] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0214] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0215] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0216] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or CD-ROM and other media that can store program codes.

[0217] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: include: Get the image of the text to be rendered; Using a multimodal model to generate text based on the image of the text to be rendered, to obtain a plurality of key points and generated texts corresponding to the plurality of key points, wherein the plurality of key points are used to represent positions in the image of the text to be rendered where corresponding generated texts need to be rendered; Using the multimodal model, the multiple key points and the generated texts corresponding to the multiple key points are processed to obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to represent an area in the image of the text to be rendered where the target generated text needs to be rendered; The target generated text is fused with the image of the text to be rendered based on the target text area to obtain a target image.

2. The method according to claim 1, characterized in that The method of generating text based on the image of the text to be rendered using the multimodal model to obtain a plurality of key points and generated text corresponding to the plurality of key points includes: Acquire first prompt information, wherein the first prompt information is used to characterize the processing task of the multimodal model, the attribute information of the multiple key points, the output format of the multiple key points, and the output format of the generated text corresponding to the multiple key points; The multimodal model is used to generate text based on the first prompt information and the image of the text to be rendered, so as to obtain the multiple key points and generated texts corresponding to the multiple key points.

3. The method according to claim 2, characterized in that The step of generating text using the multimodal model based on the first prompt information and the image of the text to be rendered to obtain the multiple key points and generated texts corresponding to the multiple key points includes: Using a text segmenter to extract features from the first prompt information to obtain text features; Using a visual encoder to extract features from the image of the text to be rendered to obtain image features; Aligning the image features with the text features using a projection layer to obtain aligned image features and aligned text features; The aligned image features and the aligned text features are input into the multimodal model, and the multiple key points output by the multimodal model and the generated texts corresponding to the multiple key points are obtained.

4. The method according to claim 1, characterized in that: The method of using the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points to obtain a target text area and a target generated text corresponding to the target text area includes: Acquire second prompt information, wherein the second prompt information is used to characterize the processing task of the multimodal model, the representation form of the target text area, the output format of the target text area, and the output format of the target generated text; The multiple key points and the generated texts corresponding to the multiple key points are processed using the multimodal model based on the second prompt information to obtain the target text area and the target generated text.

5. The method according to claim 4, characterized in that The using the multimodal model to process the multiple key points and the generated texts corresponding to the multiple key points based on the second prompt information to obtain the target text area includes: Processing the multiple key points using the multimodal model based on the second prompt information and the number of characters of the generated text corresponding to the multiple key points to obtain a Bezier curve corresponding to the target text area; The target text area is obtained based on the positions of the multiple control points of the Bezier curve.

6. The method according to claim 1, characterized in that The step of fusing the target generated text with the image of the text to be rendered based on the target text area to obtain a target image includes: Converting the target text area into target word positions, wherein the target word positions are used to represent positions of words in the target generated text in the image of the text to be rendered; Based on the target word position, the words in the target generated text are fused with the image of the text to be rendered to obtain the target image.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Outputting the multiple key points and generated texts corresponding to the multiple key points; When the first feedback result is received, the first feedback result and the first unmodified result are processed using the multimodal model to obtain the target text area and the target generated text, wherein the first feedback result is obtained by modifying the multiple key points and / or the generated texts corresponding to the multiple key points, and the first unmodified result is used to represent the results other than the first feedback result in the multiple key points and the generated texts corresponding to the multiple key points.

8. The method according to claim 7, characterized in that The method further comprises: The multimodal model is adjusted based on the image of the text to be rendered and the first feedback result.

9. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Outputting the target text area and the target generated text; When the second feedback result is received, the modified generated text is fused with the image of the text to be rendered based on the modified text area to obtain the target image, or the target generated text is fused with the image of the text to be rendered based on the modified text area to obtain the target image, or the modified generated text is fused with the image of the text to be rendered based on the target text area to obtain the target image, wherein the second feedback result is a result of modifying the target text area and / or the target generated text.

10. The method according to claim 9, characterized in that The method further comprises: The multimodal model is adjusted based on the image of the text to be rendered and the second feedback result.

11. An image processing method, characterized in that: include: Capturing scene images in real scenes; Generating text based on the scene image using a multimodal model to obtain a plurality of key points and scene texts corresponding to the plurality of key points, wherein the plurality of key points are used to represent positions of the corresponding scene texts to be generated in the scene image; Using the multimodal model, the multiple key points and the scene texts corresponding to the multiple key points are processed to obtain a target text area and a target scene text corresponding to the target text area, wherein the target text area is used to represent the area in the scene image where the target scene text is to be generated; The target scene text is fused with the scene image based on the target text area to obtain a scene text image.

12. The method according to claim 11, characterized in that The method of generating text based on the scene image using a multimodal model to obtain a plurality of key points and scene texts corresponding to the plurality of key points includes: Acquire first prompt information, wherein the first prompt information is used to characterize the processing task of the multimodal model, the attribute information of the multiple key points, the output format of the multiple key points, and the output format of the scene text corresponding to the multiple key points; The multimodal model is used to generate text based on the first prompt information and the scene image to obtain the multiple key points and the scene texts corresponding to the multiple key points.

13. The method according to claim 11, characterized in that The method of using the multimodal model to process the multiple key points and the scene texts corresponding to the multiple key points to obtain a target text area and a target scene text corresponding to the target text area includes: Acquire second prompt information, wherein the second prompt information is used to characterize a processing task of the multimodal model, a representation form of the target text region, an output format of the target text region, and an output format of the target scene text; The plurality of key points and the scene texts corresponding to the plurality of key points are processed by using the multimodal model based on the second prompt information to obtain the target text area and the target scene text.

14. An image processing method, characterized in that: include: Receive the image of the text to be rendered sent by the client; Generating text based on the image of the text to be rendered using a multimodal model to obtain a plurality of key points and generated texts corresponding to the plurality of key points, wherein the plurality of key points are used to characterize positions of the corresponding generated texts to be generated in the image of the text to be rendered; Using the multimodal model, the multiple key points and the generated texts corresponding to the multiple key points are processed to obtain a target text area and a target generated text, wherein the target text area is used to represent an area in the image of the text to be rendered where the target generated text is to be generated; The target text area and the target generated text are sent to the client.

15. An image generation method, characterized in that: include: Acquire an image of the text to be rendered by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter includes the image of the text to be rendered; Generating text based on the image of the text to be rendered using a multimodal model to obtain a plurality of key points and generated texts corresponding to the plurality of key points, wherein the plurality of key points are used to characterize positions of the corresponding generated texts to be generated in the image of the text to be rendered; Using the multimodal model, the multiple key points and the generated texts corresponding to the multiple key points are processed to obtain a target text area and a target generated text corresponding to the target text area, wherein the target text area is used to represent an area in the image of the text to be rendered where the target generated text is to be generated; Based on the target text area, the target generated text is merged with the image of the text to be rendered to obtain a target image; The target image is output by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter includes the target image.

16. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 15 when running.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 15.

18. A computer program product, characterized in that The invention comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 15.