Image conversion correction method, device and storage medium

Abstract

The application provides a kind of picture-text conversion correction method, equipment and storage medium, it is related to image processing field.The method comprises: obtaining the original image to be processed and inputting original image into the preset graph text model to generate text description;Based on original image, text description and preset text graph model, obtain simulated image;Original image and simulated image are analyzed to obtain image slice pair and the similarity data of each image slice pair;According to similarity data, the image slice pair is analyzed to generate correction text;The correction text is corrected to obtain target text by text description.Through the way of original image generating text description, text description generating simulated image, original image and simulated image are compared to correct text description, effectively reduce the object illusion in text description and the situation that text description and picture content are inconsistent, improve the accuracy of text description generated by picture-text conversion.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a method and device for correcting image-text conversion and a storage medium. BACKGROUND

[0002] An image-to-text model can convert image content into a text description. Existing technologies combine visual and other non-verbal modalities with language to develop image-to-text models that can perform a variety of complex tasks and generate comprehensive descriptions of given images. However, existing image-to-text models still have the phenomenon of generating text descriptions that are inconsistent with input images, which is referred to as "hallucination" and is widespread in existing models. Hallucination is generally divided into three categories: (1) object hallucination, i.e., adding or describing non-existent objects in the text description generated based on the image; (2) relationship hallucination, i.e., the model's incorrect description of the relationships between objects in the image; and (3) attribute hallucination, i.e., the model's failure to accurately reflect the attributes of objects in the image. The above hallucination phenomenon greatly limits the effectiveness of image-to-text models in practical applications, especially in fields such as medicine, finance, and transportation that require high precision, and hallucination can cause serious consequences.

[0003] Existing technologies have adopted a method of limiting the length of the text input data to reduce the probability of model hallucination. Although this method can reduce the occurrence of hallucination, it also results in insufficient details in the generated text description, adversely affecting the accuracy of the text description. SUMMARY

[0004] The present application provides a method and device for correcting image-text conversion and a storage medium to solve the technical problem of poor text description accuracy in the image-text conversion process in the prior art.

[0005] In a first aspect, the present application provides a method for correcting image-text conversion, comprising:

[0006] obtaining an original image to be processed and inputting the original image into a preset image-to-text model to generate a text description;

[0007] obtaining a simulated image based on the original image, the text description, and a preset text-to-image model;

[0008] performing slice analysis on the original image and the simulated image to obtain image slice pairs and similarity data for each image slice pair;

[0009] analyzing the image slice pairs based on the similarity data to generate a corrected text;

[0010] correcting the text description based on the corrected text to obtain a target text.

[0011] In a possible design, the obtaining the simulation image based on the original image, the text description and a preset text-to-image model comprises the following steps.

[0012] extracting image features from the original image to obtain a visual feature vector, the visual feature vector being used to represent visual information of the image;

[0013] performing text feature extraction on the text description to obtain a text feature vector, the text feature vector being used to represent semantic information of the text;

[0014] performing feature fusion on the visual feature vector and the text feature vector to obtain a fused feature vector;

[0015] inputting the fused feature vector into the preset text-to-image model to obtain the simulation image.

[0016] In a possible design, the performing slice analysis on the original image and the simulation image to obtain image slice pairs and similarity data of each image slice pair comprises the following steps.

[0017] performing object detection on the original image and the simulation image to obtain a plurality of noun objects;

[0018] extracting images corresponding to each noun object from the original image and the simulation image to obtain a plurality of image slice pairs;

[0019] performing similarity analysis on the images in each image slice pair to obtain similarity data.

[0020] In a possible design, the extracting images corresponding to each noun object from the original image and the simulation image to obtain a plurality of image slice pairs comprises the following steps.

[0021] extracting a bounding box of each noun object from the original image and the simulation image;

[0022] cropping corresponding image slices from the original image and the simulation image according to the bounding boxes, each image slice corresponding to one noun object;

[0023] matching the slices corresponding to the same noun object in the original image and the simulation image to obtain a plurality of image slice pairs.

[0024] In a possible design, the performing similarity analysis on the images in each image slice pair to obtain similarity data comprises the following steps.

[0025] performing feature extraction on the images in the image slice pair to obtain feature data;

[0026] perform similarity calculation on the images in the image slice pair based on the feature data to obtain similarity data.

[0027] In a possible design, the analysis on the image slice pair based on the similarity data to generate the correction text comprises:

[0028] filtering, according to the similarity data and a preset threshold, to obtain a target slice pair with similarity data less than the preset threshold;

[0029] inputting the images in the target slice pair into a preset multi-modal large language model respectively to obtain comparison text by description;

[0030] performing analysis on the comparison text based on the original image slice in the target slice pair to obtain a correction text.

[0031] In a possible design, the correction of the text description by using the correction text to obtain a target text comprises:

[0032] extracting, from the text description, to-be-corrected text corresponding to the correction text;

[0033] replacing the correction text with the to-be-corrected text to obtain a target text.

[0034] In a second aspect, the present application provides a text conversion correction device, comprising:

[0035] an acquisition module configured to acquire an original image to be processed and input the original image into a preset image-to-text model to generate a text description;

[0036] a simulation module configured to obtain a simulation image based on the original image, the text description, and a preset text-to-image model;

[0037] a slicing module configured to perform slicing analysis on the original image and the simulation image to obtain an image slice pair and similarity data of each image slice pair;

[0038] an analysis module configured to analyze the image slice pair based on the similarity data to generate a correction text;

[0039] a correction module configured to correct the text description by using the correction text to obtain a target text.

[0040] In a possible design, the simulation module is specifically configured to:

[0041] extract an image feature from the original image to obtain a visual feature vector, the visual feature vector being used to represent visual information of the image;

[0042] text feature extraction is performed on the text description to obtain a text feature vector, and the text feature vector is used to represent semantic information of the text;

[0043] feature fusion is performed on the visual feature vector and the text feature vector to obtain a fusion feature vector;

[0044] the fusion feature vector is input into the preset text-to-image model to obtain a simulation image.

[0045] In a possible design, the slicing module is specifically configured to:

[0046] object detection is performed on the original image and the simulation image to obtain a plurality of noun objects;

[0047] image extraction is performed on the original image and the simulation image to obtain a plurality of image slice pairs corresponding to each of the noun objects;

[0048] similarity analysis is performed on the images in each of the image slice pairs to obtain similarity data.

[0049] In a possible design, the slicing module is further configured to:

[0050] a bounding box of each of the noun objects is extracted from the original image and the simulation image;

[0051] an image slice corresponding to each of the noun objects is cropped from the original image and the simulation image according to the bounding box;

[0052] the same noun objects in the original image and the simulation image are matched to obtain a plurality of image slice pairs.

[0053] In a possible design, the slicing module is further configured to:

[0054] feature extraction is performed on the images in the image slice pairs to obtain feature data;

[0055] similarity calculation is performed on the images in the image slice pairs based on the feature data to obtain similarity data.

[0056] In a possible design, the analysis module is specifically configured to:

[0057] target slice pairs in which the similarity data is less than a preset threshold are screened according to the similarity data and the preset threshold;

[0058] the images in the target slice pairs are input into a preset multi-modal large language model to obtain comparison texts;

[0059] Based on the original image slice pair in the target slice pair, the contrast text is analyzed to obtain a corrected text.

[0060] In a possible design, the correction module is specifically configured to:

[0061] extract a text to be corrected corresponding to the corrected text from the text description;

[0062] replace the corrected text with the text to be corrected to obtain a target text.

[0063] In a third aspect, the present application provides a picture-text conversion correction device, comprising a processor and a memory in communication connection with the processor;

[0064] The memory stores computer execution instructions;

[0065] The processor executes the computer execution instructions stored in the memory, so that the path query device accessing the network resource executes the picture-text conversion correction method in any one of the first aspect.

[0066] In a fourth aspect, the present application provides a computer readable storage medium, and the computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the picture-text conversion correction method in any one of the first aspect.

[0067] The picture-text conversion correction method, device and storage medium provided by the present application, by obtaining the original image to be processed and inputting the original image into the preset picture-to-text model to generate a text description; based on the original image, the text description and the preset text-to-picture model, a simulation image is obtained; the original image and the simulation image are analyzed to obtain an image slice pair and similarity data of each image slice pair; the image slice pair is analyzed according to the similarity data to generate a corrected text; and the target text is obtained by correcting the text description through the corrected text. The text description is corrected by generating a text description from an original image, generating a simulation image from a text description, and comparing an original image with a simulation image. The occurrence of object illusion in the text description and inconsistency between the text description and the picture content is effectively reduced, the accuracy of the text description generated by the picture-text conversion is improved, and the text description is closer to the actual content of the original image. BRIEF DESCRIPTION OF DRAWINGS

[0068] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the present application.

[0069] Figure 1 is a flowchart of the picture-text conversion correction method provided by the embodiments of the present application Figure 1 ;

[0070] Figure 2 is a flowchart of a picture-text conversion correction method provided by an embodiment of the present application Figure 2 ;

[0071] Figure 3 is a flowchart of a picture-text conversion correction method provided by an embodiment of the present application Figure 3 ;

[0072] Figure 4 is a flowchart of a picture-text conversion correction method provided by an embodiment of the present application Figure 4 ;

[0073] Figure 5 is a flowchart of a picture-text conversion correction method provided by an embodiment of the present application Figure 5 ;

[0074] Figure 6 is a structural diagram of a picture-text conversion correction device provided by an embodiment of the present application

[0075] Figure 7 is a hardware structural diagram of a picture-text conversion correction device provided by an embodiment of the present application

[0076] The specific embodiments of the present application have been shown in the above-described drawings, and will be described in more detail hereinafter. The drawings and the written description are not intended to limit the scope of the present application in any way, but to explain the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0077] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without any creative work fall within the scope of protection of the present application.

[0078] The image-to-text model can convert image content into a text description. The prior art combines visual and other non-verbal modalities with language to develop image-to-text models that can perform a variety of complex tasks and generate comprehensive descriptions of given images. However, existing image-to-text models still have the phenomenon of generating text descriptions that are inconsistent with the input images, which is referred to as "hallucination" and is widespread in existing models. Hallucination is generally divided into three categories: (1) object hallucination, i.e., adding or describing non-existent objects in the text description generated based on the image; (2) relationship hallucination, i.e., the model incorrectly describes the relationships between objects in the image; and (3) attribute hallucination, i.e., the model fails to accurately reflect the attributes of objects in the image. The above hallucination phenomenon greatly limits the effectiveness of image-to-text models in practical applications, especially in fields such as medicine, finance, and transportation that require high precision, and hallucination can cause serious consequences.

[0079] The prior art uses a method of limiting the text length of input data to reduce the probability of model hallucination. Although this method can reduce hallucination, it also results in insufficient details in the generated text description, which adversely affects the accuracy of the text description.

[0080] The image-to-text correction method provided by the present application aims to solve the above technical problems of the prior art.

[0081] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described again in some examples. The embodiments of the present application will be described below with reference to the accompanying drawings.

[0082] Figure 1 Flowchart of the image-to-text correction method of the embodiments of the present application Figure 1 As shown in Figure 1 The present embodiment provides an image-to-text correction method, which includes the following steps:

[0083] Step S101: Obtain the original image to be processed and input the original image into a pre-set image-to-text model to generate a text description.

[0084] Specifically, the image-to-text model analyzes the original image, including but not limited to identifying various elements in the image, such as objects, people, scenes, colors, shapes, etc.; the generated text description will describe the characteristics and mutual relationships of these objects in detail, such as the specific name, quantity, state of the object, the action, expression of the person, the atmosphere of the scene, etc. Before inputting the original image into the preset image-to-text model, the image can be preprocessed, such as adjusting the image to the fixed size required by the image-to-text model to ensure consistency of input; normalizing the image pixel values, for example, scaling the pixel values to 0 to 1, which helps to improve the stability of the model analysis process; denoising the normalized image to improve image quality; converting the image to a specific format required by the image-to-text model to obtain better image-to-text conversion effect.

[0085] Step S102: obtaining a simulation image based on the original image, the text description, and the preset text-to-image model.

[0086] Specifically, the purpose of generating a simulation image is to verify the matching degree of the text description and the original image. The generated simulation image should be highly consistent with the original image in ideal cases, and the differences between the simulation image and the original image can reflect whether the text description accurately describes the content of the original image.

[0087] Step S103: performing slice analysis on the original image and the simulation image to obtain image slice pairs and similarity data of each image slice pair.

[0088] Specifically, by dividing the image into multiple slices, the differences between the original image and the simulation image can be viewed in more detail, and by performing separate slicing and similarity calculation on each noun object, the analysis is more specific and targeted.

[0089] Step S104: analyzing the image slice pairs according to the similarity data to generate a correction text.

[0090] Specifically, the similarity data helps to identify specific errors in the model when generating images, and the correction text provides specific correction content; by generating a correction text, inaccurate parts in the text description can be identified and corrected, improving the quality of image-to-text conversion.

[0091] Step S105: modifying the text description by the correction text to obtain a target text.

[0092] Specifically, by modifying the text description, the accuracy and consistency of the generated text can be significantly improved; through detailed text modification, the text description generated by the image can more accurately reflect and describe the image content, effectively improving the precision and quality of the generated content.

[0093] The image-text conversion correction method provided by the embodiment of the present application can effectively reduce the occurrence of object hallucination in the text description and inconsistency between the text description and the picture content, improve the accuracy of the text description generated by the image-text conversion, and make the text description closer to the actual content of the original image.

[0094] Figure 2 The image-text conversion correction method flowchart provided by the embodiment of the present application Figure 2 . As shown in Figure 3 , the embodiment details the process of obtaining the simulated image based on the original image, the text description, and the preset text-to-image model in the above embodiment. The specific implementation of the process includes the following steps:

[0095] Step S201: Extract image features from the original image to obtain a visual feature vector, which is used to represent the visual information of the image.

[0096] Specifically, the visual information of the image can be captured by using existing deep learning models such as convolutional neural networks (CNN).

[0097] Step S202: Perform text feature extraction on the text description to obtain a text feature vector, which is used to represent the semantic information of the text.

[0098] Specifically, the accuracy can be improved by performing preprocessing operations such as cleaning, tokenization, and removing stop words on the text description before feature extraction. Text cleaning can remove noise such as punctuation, special characters, and unnecessary spaces, which helps to simplify the text and reduce unnecessary complexity; tokenization can divide the text into more easily processed word units, facilitating subsequent analysis and processing; removing stop words can reduce the interference of irrelevant words and make key information more prominent, thereby helping to more accurately extract text features and improve accuracy and effectiveness in subsequent processing, so that the model can better understand and process text information, and ultimately generate more expected simulated images. By using existing natural language processing techniques such as word embedding and BERT, the text can be converted into a text feature vector, which represents the semantic information of the text.

[0099] Step S203: Perform feature fusion on the visual feature vector and the text feature vector to obtain a fused feature vector.

[0100] Specifically, the fusion process can dynamically adjust the weights of the visual feature vector and the text feature vector through the attention mechanism, so as to better capture the correlation between the two. The attention mechanism can flexibly allocate attention according to the context information, and by dynamically adjusting the feature weights, the image and the text can be better fused, and the generalization ability can be enhanced. According to actual needs, a plurality of fusion modes such as vector splicing, weighted average, linear transformation, etc. can be selected.

[0101] Step S204: input the fused feature vector into a preset text-to-image model to obtain a simulated image.

[0102] Specifically, the text-to-image model can select existing generative adversarial networks, diffusion models, variational autoencoders, etc. These models can generate images that meet the input description according to the input fused feature vector.

[0103] The embodiment of the application effectively combines the visual information of the image and the semantic information of the text to generate a simulated image consistent with the input description. In the case of accurate text description, the generated simulated image should be highly consistent with the original image. The purpose of generating the simulated image is to slice and compare with the original image to check for errors.

[0104] Figure 3 The text-to-image conversion correction method flowchart provided by the embodiment of the application Figure 3 As shown in Figure 4 , the embodiment details the process of slicing and analyzing the original image and the simulated image to obtain the image slice pair and the similarity data of each image slice pair. The specific implementation of the process includes the following steps:

[0105] Step S301: performing object detection on the original image and the simulated image to obtain a plurality of noun objects.

[0106] Specifically, the noun objects in the image can be recognized and located through an object detection algorithm such as YOLO (You Only Look Once), Faster R-CNN or SSD (Single Shot MultiBox Detector); the noun objects include people, objects, etc., which are labeled through a bounding box to determine their position and category in the image.

[0107] Step S302: extracting the image corresponding to each noun object from the original image and the simulated image to obtain a plurality of image slice pairs.

[0108] Specifically, the image regions corresponding to each detected noun object are extracted from the original image and the simulated image respectively to form a plurality of image slice pairs; each image slice pair contains an image slice from the original image and an image slice from the simulated image. By generating the image slice pairs, more detailed similarity analysis can be performed at the object level; the similarity analysis of the slice pairs can help analyze the generation quality of each object in the simulated image; by comparing the slice pairs, the differences between the original image and the simulated image at each noun object can be identified.

[0109] Step S303: performing similarity analysis on the images in each image slice pair to obtain similarity data.

[0110] Specifically, the similarity data can be used to determine the difference between the original image and the simulated image.

[0111] In some optional embodiments, the above step S302 includes:

[0112] Step a1: extracting the bounding box of each noun object from the original image and the simulated image.

[0113] Specifically, a bounding box is generated for each detected noun object. These bounding boxes define the position and size of the object in the image.

[0114] Step a2: cutting out the corresponding image slices from the original image and the simulated image according to the bounding boxes, each image slice corresponding to a noun object.

[0115] Specifically, according to the bounding box of each noun object, the corresponding image region is cut out from the original image to obtain a plurality of image slices; according to the bounding box of each noun object, the corresponding image region is cut out from the simulated image.

[0116] Step a3: matching the slices corresponding to the same noun object in the original image and the simulated image to obtain a plurality of image slice pairs.

[0117] Specifically, the slices corresponding to the same noun object in the original image and the simulated image are matched to form a plurality of image slice pairs; each slice pair is composed of an object slice in the original image and a corresponding object slice in the simulated image. By generating the image slice pairs, more detailed similarity analysis can be performed at the object level; the similarity analysis of the slice pairs can help view the generation quality of each object in the simulated image; by comparing the slice pairs, the differences between the original image and the simulated image at each noun object can be identified.

[0118] In some optional embodiments, the above step S303 includes:

[0119] Step b1, feature extraction is performed on the images in the image slice pair to obtain feature data.

[0120] Specifically, features are extracted from each image slice for similarity calculation, and a pre-trained convolutional neural network model can be selected for feature extraction.

[0121] Step b2, similarity calculation is performed on the images in the image slice pair based on the feature data to obtain similarity data.

[0122] Specifically, for each image slice pair, the similarity score between the feature vectors thereof is calculated by, for example, cosine similarity or Euclidean distance. The similarity data can be used to analyze the quality of the simulation image and determine the accuracy of the text description.

[0123] The embodiment of the present application can more carefully compare the differences between the original image and the simulation image by dividing the image into multiple slices, and the analysis is more specific and targeted by separately slicing and calculating the similarity of each noun object.

[0124] Figure 4 The text-image conversion correction method flowchart provided by the embodiment of the present application is shown in Figure 4 . As Figure 5 shown, the embodiment details the process of generating correction text by analyzing the image slice pair according to the similarity data in the above embodiment, and the specific implementation of the process includes the following steps:

[0125] Step S401: filtering to obtain a target slice pair with similarity data less than a preset threshold value according to the similarity data and the preset threshold value.

[0126] Specifically, a similarity threshold value (e.g., 0.8) is pre-set; then the similarity data of all image slice pairs is compared with the preset threshold value, and the slice pair with a similarity lower than the threshold value is marked as a target slice pair. The threshold value can be adjusted according to the specific application requirements and the similarity measurement method to ensure that the selected slice has a large difference.

[0127] Step S402: inputting the pictures in the target slice pair into a preset multi-modal large language model to obtain comparison text.

[0128] Specifically, for each target slice pair, the original image slice and the simulation image slice are input into the multi-modal large language model to obtain the description text generated by the model, which is used for subsequent analysis and comparison.

[0129] Step S403: analyzing the comparison text based on the original image slice in the target slice pair to obtain correction text.

[0130] Specifically, by comparing the description texts of the original image slice and the simulated image slice, the inconsistent or inaccurate places are identified; according to the true features of the original image slice, the description text of the simulated image slice is corrected to generate a corrected text. The corrected text can correctly describe the object features of the slice position corresponding to the original image.

[0131] By identifying and correcting the image slices with low similarity, the embodiment of the present application can re-analyze the slices with larger differences to generate new text descriptions as corrected texts. The similarity data help to identify the specific errors of the model when generating images, and the corrected texts provide specific correction contents; by generating the corrected texts, the inaccurate places in the text description can be identified and corrected, and the quality of the image-text conversion is improved.

[0132] Figure 5 The image-text conversion correction method flowchart provided by the embodiment of the present application is shown in Figure 5 . As shown in Figure 6 , the embodiment details the process of obtaining the target text by correcting the text description with the corrected text in the above embodiment. The specific implementation of the process includes the following steps:

[0133] Step S501: Extracting the to-be-corrected text corresponding to the corrected text from the text description.

[0134] Specifically, the position of the to-be-corrected text in the text description is accurately located and extracted by the above-mentioned noun object.

[0135] Step S502: Replacing the corrected text with the to-be-corrected text to obtain the target text.

[0136] Specifically, the to-be-corrected text in the initial text description is replaced with the corrected text. This step ensures that the text description more accurately reflects the content of the original image

[0137] The embodiment of the present application can significantly improve the accuracy and consistency of the generated text by correcting the text description; through detailed text correction, the text description of the image generation can more accurately reflect and describe the image content, effectively improving the precision and quality of the generated content.

[0138] Figure 6 The structure diagram of the image-text conversion correction device provided by the embodiment of the present application is shown in Figure 6 . As shown in the figure, the image-text conversion correction device 60 includes an acquisition module 601, a simulation module 602, a slice module 603, an analysis module 604, and a correction module 605. Wherein:

[0139] The acquisition module 601 is configured to acquire the original image to be processed and input the original image into a preset image-to-text model to generate a text description;

[0140] The simulation module 602 is configured to obtain a simulation image based on the original image, the text description, and a preset text-to-image model.

[0141] The slicing module 603 is configured to perform slicing analysis on the original image and the simulation image to obtain an image slice pair and similarity data of each image slice pair.

[0142] The analysis module 604 is configured to analyze the image slice pair according to the similarity data to generate a correction text.

[0143] The correction module 605 is configured to correct the text description by using the correction text to obtain a target text.

[0144] In a possible design, the simulation module 602 is specifically configured to:

[0145] extract an image feature from the original image to obtain a visual feature vector, the visual feature vector being used to represent visual information of the image;

[0146] perform text feature extraction on the text description to obtain a text feature vector, the text feature vector being used to represent semantic information of the text;

[0147] perform feature fusion on the visual feature vector and the text feature vector to obtain a fused feature vector;

[0148] input the fused feature vector into the preset text-to-image model to obtain the simulation image.

[0149] In a possible design, the slicing module 603 is specifically configured to:

[0150] perform object detection on the original image and the simulation image to obtain a plurality of noun objects;

[0151] extract, from the original image and the simulation image, an image corresponding to each of the noun objects to obtain a plurality of image slice pairs;

[0152] perform similarity analysis on the images in each of the image slice pairs to obtain similarity data.

[0153] In a possible design, the slicing module 603 is further configured to:

[0154] extract a bounding box of each of the noun objects from the original image and the simulation image;

[0155] cut out, according to the bounding box, a corresponding image slice from the original image and the simulation image respectively, each of the image slices corresponding to one of the noun objects;

[0156] match the slices corresponding to the same noun object in the original image and the simulation image to obtain the plurality of image slice pairs.

[0157] In a possible design, the slicing module 603 is further configured to:

[0158] perform feature extraction on the images in the image slice pair to obtain feature data;

[0159] perform similarity calculation on the images in the image slice pair based on the feature data to obtain similarity data.

[0160] In a possible design, the analysis module 604 is specifically configured to:

[0161] filter, according to the similarity data and a preset threshold, to obtain a target slice pair with similarity data less than the preset threshold;

[0162] input the images in the target slice pair into a preset multi-modal large language model respectively to obtain comparison texts by description;

[0163] analyze the comparison texts based on the original image slice pair in the target slice pair, to obtain correction texts.

[0164] In a possible design, the correction module 605 is specifically configured to:

[0165] extract, from the text description, to-be-corrected texts corresponding to the correction texts;

[0166] replace the correction texts with the to-be-corrected texts to obtain target texts.

[0167] The image-text conversion correction device provided in this embodiment can be used to execute the image-text conversion correction method described above, and has similar implementation principles and technical effects, which will not be described here again in this embodiment.

[0168] Figure 7 A hardware structure diagram of the image-text conversion correction device provided in this embodiment is shown in FIG. 7, which includes at least one processor 701 and a memory 702. Optionally, the image-text conversion correction device 70 further includes a communication component 703. The processor 701, the memory 702, and the communication component 703 are connected through a bus 704. Figure 7

[0169] In the specific implementation process, the at least one processor 701 executes the computer execution instructions stored in the memory 702, so that the at least one processor 701 executes the image-text conversion correction method as described above.

[0170] The communication component 703 can interact with a server.

[0171] The specific implementation process of the processor 701 can refer to the method embodiments described above, which has similar implementation principles and technical effects, and will not be described here again in this embodiment.

[0172] In the above​Figure 7 In the illustrated embodiment, it is to be understood that the processor can be a central processing unit (CPU), and can also be other general purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in conjunction with the application can be directly embodied as hardware processor execution, or executed by a combination of hardware and software modules in the processor.

[0173] The memory can include a high-speed RAM memory, and can also include a non-volatile storage NVM, such as at least one disk memory.

[0174] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0175] The present application also provides a computer readable storage medium, the computer readable storage medium stores computer execution instructions, when the processor executes the computer execution instructions, the method for converting and correcting image and text as described above is realized.

[0176] The computer readable storage medium described above, the readable storage medium can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0177] An example readable storage medium is coupled to the processor such that the processor can read information from the readable storage medium and can write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0178] The division of the units is only a logical function division, and in actual implementation, another division manner can be used, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units or components shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0179] The units described above as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, can be located in one place or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0180] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0181] If the above functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0182] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. The program executes to perform the steps of the above-mentioned method embodiments; and the foregoing storage medium includes various storage media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method of modifying a text-to-speech conversion, characterized by, The method comprises the following steps: acquire an original image to be processed and input the original image into a preset image-to-text model to generate a text description; obtain a simulated image based on the original image, the text description, and a preset text-to-image model; perform object detection on the original image and the simulated image to obtain a plurality of noun objects; extract an image corresponding to each noun object from the original image and the simulated image to obtain a plurality of image slice pairs; and perform similarity analysis on the images in each image slice pair to obtain similarity data; filter the target slice pair whose similarity data is less than a preset threshold value according to the similarity data and the preset threshold value; and input the images in the target slice pair into a preset multi-modal large language model to generate a comparison text; analyze the comparison text based on the original image slice in the target slice pair to obtain a correction text; correct the text description based on the correction text to obtain a target text.

2. The method of claim 1, wherein, The method comprises the following steps: extract image features from the original image to obtain a visual feature vector, which is used to represent visual information of the image; extract text features from the text description to obtain a text feature vector, which is used to represent semantic information of the text; fuse the visual feature vector and the text feature vector to obtain a fused feature vector; input the fused feature vector into the preset text-to-image model to obtain a simulated image.

3. The method of claim 1, wherein, The method comprises the following steps: extract a bounding box of each noun object from the original image and the simulated image; cut out a corresponding image slice from the original image and the simulated image according to the bounding box, with each image slice corresponding to a noun object; match the slices corresponding to the same noun object in the original image and the simulated image to obtain a plurality of image slice pairs.

4. The method of claim 1, wherein, The method comprises the following steps: extract features from the images in the image slice pair to obtain feature data; calculate the similarity of the images in the image slice pair based on the feature data to obtain similarity data.

5. The method of claim 1, wherein, The method comprises the following steps: extract the text to be corrected corresponding to the correction text from the text description; replace the correction text with the text to be corrected to obtain a target text.

6. An image processing apparatus, characterized by comprising: The method comprises the following steps: an acquisition module, configured to acquire an original image to be processed and input the original image into a preset image-to-text model to generate a text description; a simulation module, configured to obtain a simulated image based on the original image, the text description, and a preset text-to-image model; a slice module, configured to perform object detection on the original image and the simulated image to obtain a plurality of noun objects; A plurality of image slice pairs are obtained by extracting images corresponding to each of the noun objects from the original image and the simulation image respectively; similarity analysis is performed on the images in each of the image slice pairs to obtain similarity data; The analysis module is configured to filter target slice pairs with similarity data less than a preset threshold according to the similarity data and the preset threshold; and input the images in the target slice pairs into a preset multi-modal large language model respectively to obtain comparison texts by description; The comparison texts are analyzed based on the original image slice pair in the target slice pair to obtain correction texts; The correction module is configured to correct the text description by using the correction texts to obtain target texts.

7. A text-image conversion and correction device, characterized in that, The method comprises: a processor and a memory connected to the processor in communication; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory to implement the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer execution instructions, and the computer execution instructions are executed by the processor to implement the method of any one of claims 1 to 5.

Images