Image-text conversion correction method and device and storage medium

Abstract

The invention provides an image-text conversion correction method and device and a storage medium, and relates to the field of image processing. The method comprises the steps of obtaining a to-be-processed original image and inputting the original image into a preset graph-to-text model to generate text description; obtaining a simulation image based on the original image, the text description and a preset text graph model; performing slice analysis on the original image and the analog image to obtain image slice pairs and similarity data of each image slice pair; analyzing the image slice pair according to the similarity data to generate a corrected text; and correcting the text description through the correction text to obtain a target text. The text description is corrected by generating the text description through the original image, generating the analog image through the text description and comparing the original image with the analog image, so that the situations that object illusion exists in the text description and the text description is inconsistent with picture content are effectively reduced, and the accuracy of the text description generated through image-text conversion is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a method, device and storage medium for image-text conversion correction. Background Art

[0002] Image-to-text models can convert image content into text descriptions. Existing technologies combine non-linguistic modalities such as vision with language to develop image-to-text models that can perform a variety of complex tasks and generate a comprehensive description of a given image. However, existing image-to-text models still have the problem of generating text descriptions that are inconsistent with the input image. This phenomenon is called "hallucination" and is widely present in existing models. Hallucinations are generally divided into three categories: (1) object hallucinations, that is, non-existent objects are mistakenly added or described in the text description generated based on the image; (2) relationship hallucinations, that is, the model incorrectly describes the relationship between the objects in the image; (3) attribute hallucinations, that is, the model fails to accurately reflect the attributes of the objects in the image. The above hallucination phenomena greatly limit the effectiveness of image-to-text models in practical applications, especially in fields that require high precision such as medicine, finance, and transportation, where hallucination phenomena may lead to serious consequences.

[0003] The prior art has adopted a method of limiting the length of the text of the input data to reduce the probability of the model generating hallucinations. Although this method can reduce the occurrence of hallucinations, it also leads to insufficient details in the generated text description, which has an adverse impact on the accuracy of the text description. Summary of the invention

[0004] The present application provides a method, device and storage medium for correcting image-text conversion, which are used to solve the technical problem of poor text description accuracy in the image-text conversion process in the prior art.

[0005] In a first aspect, the present application provides a method for image-text conversion correction, comprising:

[0006] Acquire an original image to be processed and input the original image into a preset image-to-text model to generate a text description;

[0007] Obtaining a simulated image based on the original image, the text description and a preset text image model;

[0008] Performing slice analysis on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair;

[0009] Analyzing the image slice pair according to the similarity data to generate corrected text;

[0010] The text description is modified by the correction text to obtain the target text.

[0011] In a possible design, obtaining a simulated image based on the original image, the text description and a preset text image model includes:

[0012] Extracting image features from the original image to obtain a visual feature vector, wherein the visual feature vector is used to represent visual information of the image;

[0013] Performing text feature extraction on the text description to obtain a text feature vector, wherein the text feature vector is used to represent semantic information of the text;

[0014] Performing feature fusion on the visual feature vector and the text feature vector to obtain a fused feature vector;

[0015] The fused feature vector is input into the preset Wensheng graph model to obtain a simulated image.

[0016] In a possible design, the performing slice analysis on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair includes:

[0017] Performing object detection on the original image and the simulated image to obtain a plurality of noun objects;

[0018] Extracting images corresponding to each of the noun objects from the original image and the simulated image to obtain a plurality of image slice pairs;

[0019] Similarity data is obtained by performing similarity analysis on the images in each of the image slice pairs.

[0020] In a possible design, extracting an image corresponding to each noun object from the original image and the simulated image to obtain a plurality of image slice pairs includes:

[0021] extracting a bounding box of each of the noun objects from the original image and the simulated image;

[0022] Cutting out corresponding image slices from the original image and the simulated image according to the bounding box, each image slice corresponding to a noun object;

[0023] The slices corresponding to the same noun object in the original image and the simulated image are matched to obtain multiple image slice pairs.

[0024] In a possible design, performing similarity analysis on the images in each of the image slice pairs to obtain similarity data includes:

[0025] Extracting features from the image in the image slice pair to obtain feature data;

[0026] Similarity data is obtained by performing similarity calculation on the images in the image slice pair based on the feature data.

[0027] In a possible design, analyzing the image slice pair according to the similarity data to generate corrected text includes:

[0028] According to the similarity data and a preset threshold, a target slice pair whose similarity data is less than the preset threshold is obtained by screening;

[0029] Inputting the images in the target slice pair into a preset multimodal large language model for description to obtain comparative text;

[0030] The comparison text is analyzed based on the original image slices in the target slice pair to obtain a corrected text.

[0031] In a possible design, the step of correcting the text description by using the correction text to obtain the target text includes:

[0032] Extracting the text to be corrected corresponding to the corrected text from the text description;

[0033] The corrected text is replaced with the text to be corrected to obtain a target text.

[0034] In a second aspect, the present application provides a device for image-text conversion correction, comprising:

[0035] An acquisition module, used for acquiring an original image to be processed and inputting the original image into a preset image-to-text model to generate a text description;

[0036] A simulation module, used for obtaining a simulated image based on the original image, the text description and a preset text image model;

[0037] A slicing module, used for performing slicing analysis on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair;

[0038] An analysis module, configured to analyze the image slice pair according to the similarity data to generate a corrected text;

[0039] The correction module is used to correct the text description by using the correction text to obtain the target text.

[0040] In one possible design, the analog module is specifically used to:

[0041] Extracting image features from the original image to obtain a visual feature vector, wherein the visual feature vector is used to represent visual information of the image;

[0042] Performing text feature extraction on the text description to obtain a text feature vector, wherein the text feature vector is used to represent semantic information of the text;

[0043] Performing feature fusion on the visual feature vector and the text feature vector to obtain a fused feature vector;

[0044] The fused feature vector is input into the preset Wensheng graph model to obtain a simulated image.

[0045] In a possible design, the slice module is specifically used to:

[0046] Performing object detection on the original image and the simulated image to obtain a plurality of noun objects;

[0047] Extracting images corresponding to each of the noun objects from the original image and the simulated image to obtain a plurality of image slice pairs;

[0048] Similarity data is obtained by performing similarity analysis on the images in each of the image slice pairs.

[0049] In one possible design, the slice module is also used to:

[0050] extracting a bounding box of each of the noun objects from the original image and the simulated image;

[0051] Cutting out corresponding image slices from the original image and the simulated image according to the bounding box, each image slice corresponding to a noun object;

[0052] The slices corresponding to the same noun object in the original image and the simulated image are matched to obtain multiple image slice pairs.

[0053] In one possible design, the slice module is also used to:

[0054] Extracting features from the image in the image slice pair to obtain feature data;

[0055] Similarity data is obtained by performing similarity calculation on the images in the image slice pair based on the feature data.

[0056] In a possible design, the analysis module is specifically used to:

[0057] According to the similarity data and a preset threshold, a target slice pair whose similarity data is less than the preset threshold is obtained by screening;

[0058] Inputting the images in the target slice pair into a preset multimodal large language model for description to obtain comparative text;

[0059] The comparison text is analyzed based on the original image slices in the target slice pair to obtain a corrected text.

[0060] In a possible design, the correction module is specifically used to:

[0061] Extracting the text to be corrected corresponding to the corrected text from the text description;

[0062] The corrected text is replaced with the text to be corrected to obtain a target text.

[0063] In a third aspect, the present application provides a device for image-text conversion and correction, comprising: a processor, and a memory communicatively connected to the processor;

[0064] Memory stores computer-executable instructions;

[0065] The processor executes the computer-executable instructions stored in the memory, so that the path query device accessing the network resource executes any one of the image-text conversion correction methods in the first aspect.

[0066] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the image-text conversion correction method as described in any one of the first aspects.

[0067] The image-text conversion correction method, device and storage medium provided by the present application generate text description by acquiring the original image to be processed and inputting the original image into a preset image-text model; obtaining a simulated image based on the original image, text description and the preset text-image model; performing slice analysis on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair; analyzing the image slice pairs according to the similarity data to generate corrected text; and correcting the text description by correcting the text to obtain the target text. The text description is corrected by generating a text description from the original image, generating a simulated image from the text description, and comparing the original image with the simulated image, which effectively reduces the occurrence of object illusions in the text description and the inconsistency between the text description and the image content, improves the accuracy of the text description generated by the image-text conversion, and makes the text description closer to the actual content of the original image. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0069] Figure 1 This is a flowchart of the image-text conversion correction method provided in the embodiment of the present application. Figure 1 ;

[0070] Figure 2 This is a flowchart of the image-text conversion correction method provided in the embodiment of the present application. Figure 2 ;

[0071] Figure 3 This is a flowchart of the image-text conversion correction method provided in the embodiment of the present application. Figure 3 ;

[0072] Figure 4 This is a flowchart of the image-text conversion correction method provided in the embodiment of the present application. Figure 4 ;

[0073] Figure 5 This is a flowchart of the image-text conversion correction method provided in the embodiment of the present application. Figure 5 ;

[0074] Figure 6 It is a structural schematic diagram of a graphic-text conversion and correction device provided in an embodiment of the present application;

[0075] Figure 7 It is a schematic diagram of the hardware structure of the image-text conversion and correction device provided in the embodiment of the present application.

[0076] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0077] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Image-to-text models can convert image content into text descriptions. Existing technologies combine non-linguistic modalities such as vision with language to develop image-to-text models that can perform a variety of complex tasks and generate a comprehensive description of a given image. However, existing image-to-text models still have the problem of generating text descriptions that are inconsistent with the input image. This phenomenon is called "hallucination" and is widely present in existing models. Hallucinations are generally divided into three categories: (1) object hallucinations, that is, non-existent objects are mistakenly added or described in the text description generated based on the image; (2) relationship hallucinations, that is, the model incorrectly describes the relationship between the objects in the image; (3) attribute hallucinations, that is, the model fails to accurately reflect the attributes of the objects in the image. The above hallucination phenomena greatly limit the effectiveness of image-to-text models in practical applications, especially in fields that require high precision such as medicine, finance, and transportation, where hallucination phenomena may lead to serious consequences.

[0079] The prior art has adopted a method of limiting the length of the text of the input data to reduce the probability of the model generating hallucinations. Although this method can reduce the occurrence of hallucinations, it also leads to insufficient details in the generated text description, which has an adverse impact on the accuracy of the text description.

[0080] The image-text conversion correction method provided in this application is intended to solve the above technical problems of the prior art.

[0081] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0082] Figure 1 Flowchart of the image-text conversion correction method according to an embodiment of the present invention Figure 1 ,like Figure 1 As shown, this embodiment provides a method for correcting image-text conversion, and the process includes the following steps:

[0083] Step S101: obtaining an original image to be processed and inputting the original image into a preset image-to-text model to generate a text description.

[0084] Specifically, the image-to-text model will analyze the original image, including but not limited to identifying various elements in the image, such as objects, people, scenes, colors, shapes, etc.; the generated text description will describe the characteristics and relationships of these objects in detail, such as the specific name, quantity, and state of the object, the actions and expressions of the characters, the atmosphere of the scene, etc. Before the original image is input into the preset image-to-text model, the image can be preprocessed, such as: adjusting the image to the fixed size required by the image-to-text model to ensure the consistency of the input; normalizing the image pixel values, such as scaling the pixel values ​​to between 0 and 1, which helps to improve the stability of the model analysis process; denoising the normalized image to improve the image quality; converting the image to the specific format required by the image-to-text model to obtain a better image-to-text conversion effect.

[0085] Step S102: obtaining a simulated image based on the original image, the text description and the preset text-based image model.

[0086] Specifically, the purpose of generating a simulated image is to verify the matching degree between the text description and the original image. Ideally, the generated simulated image should maintain a high degree of consistency with the original image, and the difference between the simulated image and the original image can reflect whether the text description accurately describes the content of the original image.

[0087] Step S103: performing slice analysis on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair.

[0088] Specifically, by dividing the image into multiple slices, the difference between the original image and the simulated image can be viewed in more detail. At the same time, by performing separate slicing and similarity calculations on each noun object, the analysis is made more specific and targeted.

[0089] Step S104: Analyze the image slice pairs according to the similarity data to generate corrected text.

[0090] Specifically, similarity data helps identify specific errors made by the model when generating images, while correction text provides specific correction content; by generating correction text, inaccuracies in text descriptions can be identified and corrected, thereby improving the quality of image-text conversion.

[0091] Step S105: Correct the text description by correcting the text to obtain the target text.

[0092] Specifically, by correcting the text description, the accuracy and consistency of the generated text can be significantly improved; through meticulous text correction, the text description generated by the image can more accurately reflect and describe the image content, effectively improving the accuracy and quality of the generated content.

[0093] The image-text conversion correction method provided by the embodiment of the present invention corrects the text description by generating a text description from an original image, generating a simulated image from the text description, and comparing the original image with the simulated image. This effectively reduces the occurrence of object illusions in the text description and the inconsistency between the text description and the image content, improves the accuracy of the text description generated by the image-text conversion, and makes the text description closer to the actual content of the original image.

[0094] Figure 2 Schematic diagram of the image-text conversion correction method provided in the embodiment of the present application Figure 2 .like Figure 2 As shown, this embodiment describes in detail the process of obtaining a simulated image based on the original image, text description and the preset text image model in the above embodiment. The specific implementation method of the process includes the following steps:

[0095] Step S201: extracting image features from the original image to obtain a visual feature vector, where the visual feature vector is used to represent visual information of the image.

[0096] Specifically, the visual information of an image can be captured through existing deep learning models such as convolutional neural networks (CNN).

[0097] Step S202: extracting text features from the text description to obtain a text feature vector, where the text feature vector is used to represent semantic information of the text.

[0098] Specifically, before feature extraction, the accuracy can be improved by preprocessing the text description, such as cleaning, word segmentation, and removing stop words. Text cleaning can remove noise in the text, such as punctuation, special characters, and extra spaces, which helps to simplify the text and reduce unnecessary complexity; word segmentation can divide the text into easier-to-process word units, facilitating subsequent analysis and processing; removing stop words can reduce the interference of irrelevant words and make key information more prominent, which helps to extract text features more accurately, thereby improving accuracy and effect in subsequent processing, enabling the model to better understand and process text information, and ultimately generate a more expected simulation image. By using existing natural language processing technologies, such as word embedding, BERT, etc., text can be converted into text feature vectors, which represent the semantic information of the text.

[0099] Step S203: Fusing the visual feature vector and the text feature vector to obtain a fused feature vector.

[0100] Specifically, the fusion process can dynamically adjust the weights of the visual feature vector and the text feature vector through the attention mechanism, so as to better capture the correlation between the two. The attention mechanism can flexibly allocate attention according to context information. By dynamically adjusting the feature weights, it can better fuse images and texts and enhance their generalization ability. You can also choose a variety of fusion methods such as vector splicing, weighted averaging, and linear transformation according to actual needs.

[0101] Step S204: inputting the fused feature vector into a preset text image model to obtain a simulated image.

[0102] Specifically, the Wensheng graph model can select existing generative adversarial networks, diffusion models, variational autoencoders, etc. These models can generate images that meet the input description based on the input fusion feature vector.

[0103] The embodiment of the present invention effectively combines the visual information of the image and the semantic information of the text to generate a simulated image that is consistent with the input description. When the text description is relatively accurate, the generated simulated image should be highly consistent with the original image. The purpose of generating the simulated image is to compare the slices with the original image to check whether there are any errors.

[0104] Figure 3 Schematic diagram of the image-text conversion correction method provided in the embodiment of the present application Figure 3 .like Figure 3 As shown, this embodiment describes in detail the process of performing slice analysis on the original image and the simulated image in the above embodiment to obtain image slice pairs and similarity data of each image slice pair. The specific implementation method of the process includes the following steps:

[0105] Step S301: performing object detection on the original image and the simulated image to obtain a plurality of noun objects.

[0106] Specifically, object detection algorithms such as YOLO (You Only Look Once), Faster R-CNN or SSD (Single Shot MultiBox Detector) can be used to identify and locate noun objects in an image. Noun objects include people, objects, etc., and their positions and categories in the image can be determined by calibrating bounding boxes.

[0107] Step S302: extracting the image corresponding to each noun object from the original image and the simulated image to obtain a plurality of image slice pairs.

[0108] Specifically, the image regions corresponding to each detected noun object are extracted from the original image and the simulated image, respectively, to form multiple image slice pairs; each image slice pair contains an image slice from the original image and an image slice from the simulated image. By generating image slice pairs, a more detailed similarity analysis can be performed at the object level; similarity analysis of slice pairs can help analyze the generation quality of each object in the simulated image; by comparing slice pairs, the differences between the original image and the simulated image for each noun object can be identified.

[0109] Step S303: performing similarity analysis on the images in each image slice pair to obtain similarity data.

[0110] Specifically, the difference between the original image and the simulated image can be determined through the similarity data.

[0111] In some optional implementations, the above step S302 includes:

[0112] Step a1, extract the bounding box of each noun object from the original image and the simulated image.

[0113] Specifically, bounding boxes are generated for each detected noun object. These bounding boxes define the location and size of the object in the image.

[0114] In step a2, corresponding image slices are cropped from the original image and the simulated image according to the bounding box, and each image slice corresponds to a noun object.

[0115] Specifically, according to the bounding box of each noun object, a corresponding image region is cropped from the original image to obtain a plurality of image slices; according to the bounding box of each noun object, a corresponding image region is cropped from the simulated image.

[0116] Step a3, matching the slices corresponding to the same noun object in the original image and the simulated image to obtain a plurality of image slice pairs.

[0117] Specifically, the slices corresponding to the same noun object in the original image and the simulated image are matched to form multiple image slice pairs; each slice pair consists of an object slice in the original image and a corresponding object slice in the simulated image. By generating image slice pairs, a more detailed similarity analysis can be performed at the object level; the similarity analysis of slice pairs can help to view the generation quality of each object in the simulated image; by comparing slice pairs, the differences between the original image and the simulated image for each noun object can be identified.

[0118] In some optional implementations, the above step S303 includes:

[0119] Step b1, extracting features from the image in the image slice pair to obtain feature data.

[0120] Specifically, features are extracted from each image slice in order to perform similarity calculation, and a pre-trained convolutional neural network model may be selected for feature extraction;

[0121] Step b2: performing similarity calculation on the images in the image slice pair based on the feature data to obtain similarity data.

[0122] Specifically, for each pair of image slices, the similarity score between their feature vectors is calculated by, for example, cosine similarity or Euclidean distance. The similarity data can be used to analyze the quality of simulated images and determine the accuracy of text descriptions.

[0123] The embodiment of the present invention can compare the difference between the original image and the simulated image in more detail by dividing the image into multiple slices. At the same time, by performing separate slicing and similarity calculation on each noun object, the analysis is more specific and targeted.

[0124] Figure 4 Schematic diagram of the image-text conversion correction method provided in the embodiment of the present application Figure 4 .like Figure 4 As shown, this embodiment describes in detail the process of analyzing the image slice pairs according to the similarity data in the above embodiment to generate the corrected text. The specific implementation of the process includes the following steps:

[0125] Step S401: Filtering, according to the similarity data and the preset threshold, to obtain target slice pairs whose similarity data is less than the preset threshold.

[0126] Specifically, a similarity threshold (e.g., 0.8) is pre-set; then the similarity data of all image slice pairs are compared with the preset threshold, and the slice pairs with similarity lower than the threshold are marked as target slice pairs. The selection of the threshold can be adjusted according to specific application requirements and similarity measurement methods to ensure that the selected slices have large differences.

[0127] Step S402: input the images in the target slice pair into a preset multimodal large language model for description to obtain comparative text.

[0128] Specifically, for each target slice pair, the original image slice and the simulated image slice are respectively input into the multimodal large language model to obtain the description text generated by the model for subsequent analysis and comparison.

[0129] Step S403: Analyze the contrast text based on the original image slice pair in the target slice pair to obtain the corrected text.

[0130] Specifically, the description texts of the original image slices and the simulated image slices are compared to identify the inconsistencies or inaccuracies in the descriptions; the description texts of the simulated image slices are corrected according to the real features of the original image slices to generate corrected texts. The corrected texts can correctly describe the object features of the original image corresponding to the slice position.

[0131] By identifying and correcting image slices with low similarity, the embodiments of the present invention can re-analyze the slice pairs with large differences in a targeted manner and generate new text descriptions as correction texts. The similarity data helps identify the specific errors of the model when generating images, while the correction text provides specific correction content; by generating the correction text, inaccuracies in the text description can be identified and corrected, thereby improving the quality of image-text conversion.

[0132] Figure 5 Schematic diagram of the image-text conversion correction method provided in the embodiment of the present application Figure 5 .like Figure 5 As shown, this embodiment describes in detail the process of correcting the text description by correcting the text in the above embodiment to obtain the target text. The specific implementation method of the process includes the following steps:

[0133] Step S501: extracting the text to be corrected corresponding to the corrected text from the text description.

[0134] Specifically, the position of the text to be corrected in the text description is accurately located and extracted through the above-mentioned noun object.

[0135] Step S502: Replace the corrected text with the text to be corrected to obtain the target text.

[0136] Specifically, the text to be corrected in the initial text description is replaced with the corrected text. This step ensures that the text description more accurately reflects the content of the original image.

[0137] The embodiments of the present invention can significantly improve the accuracy and consistency of generated text by correcting text descriptions; through careful text correction, the text description generated by an image can more accurately reflect and describe the image content, effectively improving the accuracy and quality of the generated content.

[0138] Figure 6 The structure diagram of the image-text conversion and correction device provided by the embodiment of the present invention is shown in FIG. Figure 6 As shown, the image-text conversion and correction device 60 includes: an acquisition module 601, a simulation module 602, a slicing module 603, an analysis module 604, and a correction module 605. Among them:

[0139] An acquisition module 601 is used to acquire an original image to be processed and input the original image into a preset image-to-text model to generate a text description;

[0140] A simulation module 602, used to obtain a simulated image based on the original image, the text description and the preset text image model;

[0141] A slicing module 603 is used to perform slicing analysis on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair;

[0142] An analysis module 604 is used to analyze the image slice pair according to the similarity data to generate a corrected text;

[0143] The correction module 605 is used to correct the text description by correcting the text to obtain the target text.

[0144] In one possible design, the simulation module 602 is specifically configured to:

[0145] Extracting image features from the original image to obtain a visual feature vector, which is used to represent the visual information of the image;

[0146] Extract text features from the text description to obtain a text feature vector, which is used to represent the semantic information of the text;

[0147] Perform feature fusion on the visual feature vector and the text feature vector to obtain a fused feature vector;

[0148] The fused feature vector is input into the preset Wensheng graph model to obtain a simulated image.

[0149] In a possible design, the slicing module 603 is specifically configured to:

[0150] Performing object detection on the original image and the simulated image to obtain multiple noun objects;

[0151] Extracting images corresponding to each noun object from the original image and the simulated image to obtain multiple image slice pairs;

[0152] Similarity data is obtained by performing similarity analysis on the images in each image slice pair.

[0153] In a possible design, the slicing module 603 is further configured to:

[0154] Extract bounding boxes of each noun object from the original and simulated images;

[0155] According to the bounding box, corresponding image slices are cropped from the original image and the simulated image respectively, and each image slice corresponds to a noun object;

[0156] The slices corresponding to the same noun object in the original image and the simulated image are matched to obtain multiple image slice pairs.

[0157] In a possible design, the slicing module 603 is further configured to:

[0158] Extracting features from the image in the image slice pair to obtain feature data;

[0159] Similarity data is obtained by performing similarity calculation on images in the image slice pair based on the feature data.

[0160] In one possible design, the analysis module 604 is specifically configured to:

[0161] According to the similarity data and the preset threshold, a target slice pair whose similarity data is less than the preset threshold is obtained by screening;

[0162] The images in the target slice pair are respectively input into the preset multimodal large language model for description to obtain comparative text;

[0163] The contrast text is analyzed based on the original image slice pair in the target slice pair to obtain the corrected text.

[0164] In one possible design, the correction module 605 is specifically configured to:

[0165] Extracting the text to be corrected corresponding to the corrected text from the text description;

[0166] The corrected text is replaced with the text to be corrected to obtain the target text.

[0167] The image-text conversion correction device provided in this embodiment can be used to execute the above-mentioned image-text conversion correction method. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0168] Figure 7 A schematic diagram of the hardware structure of the image-text conversion and correction device provided by an embodiment of the present invention, such as Figure 7 As shown, the image-text conversion and correction device 70 includes: at least one processor 701 and a memory 702. Optionally, the image-text conversion and correction device 70 also includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0169] In a specific implementation process, at least one processor 701 executes the computer execution instructions stored in the memory 702, so that at least one processor 701 executes the above-mentioned image-text conversion correction method.

[0170] The communication component 703 can exchange data with the server.

[0171] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0172] In the above Figure 7 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0173] The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk storage.

[0174] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0175] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above-mentioned image-text conversion correction method is implemented.

[0176] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0177] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0178] The division of the above units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0179] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0181] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0182] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for correcting image-text conversion, characterized in that: include: Acquire an original image to be processed and input the original image into a preset image-to-text model to generate a text description; Obtaining a simulated image based on the original image, the text description and a preset text image model; Performing slice analysis on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair; Analyzing the image slice pair according to the similarity data to generate corrected text; The text description is modified by the correction text to obtain the target text.

2. The method according to claim 1, characterized in that The step of obtaining a simulated image based on the original image, the text description and a preset text image model includes: Extracting image features from the original image to obtain a visual feature vector, wherein the visual feature vector is used to represent visual information of the image; Performing text feature extraction on the text description to obtain a text feature vector, wherein the text feature vector is used to represent semantic information of the text; Performing feature fusion on the visual feature vector and the text feature vector to obtain a fused feature vector; The fused feature vector is input into the preset Wensheng graph model to obtain a simulated image.

3. The method according to claim 1, characterized in that Slice analysis is performed on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair, including: Performing object detection on the original image and the simulated image to obtain a plurality of noun objects; Extracting images corresponding to each of the noun objects from the original image and the simulated image to obtain a plurality of image slice pairs; Similarity data is obtained by performing similarity analysis on the images in each of the image slice pairs.

4. The method according to claim 3, characterized in that Extracting images corresponding to each of the noun objects from the original image and the simulated image to obtain a plurality of image slice pairs includes: extracting a bounding box of each of the noun objects from the original image and the simulated image; Cutting out corresponding image slices from the original image and the simulated image according to the bounding box, each image slice corresponding to a noun object; The slices corresponding to the same noun object in the original image and the simulated image are matched to obtain multiple image slice pairs.

5. The method according to claim 3, characterized in that: The performing similarity analysis on the images in each of the image slice pairs to obtain similarity data comprises: Extracting features from the image in the image slice pair to obtain feature data; Similarity data is obtained by performing similarity calculation on the images in the image slice pair based on the feature data.

6. The method according to claim 1, characterized in that The step of analyzing the image slice pair according to the similarity data to generate the corrected text comprises: According to the similarity data and a preset threshold, a target slice pair whose similarity data is less than the preset threshold is obtained by screening; Inputting the images in the target slice pair into a preset multimodal large language model for description to obtain comparative text; The comparison text is analyzed based on the original image slices in the target slice pair to obtain a corrected text.

7. The method according to claim 1, characterized in that The step of correcting the text description by using the correction text to obtain the target text includes: Extracting the text to be corrected corresponding to the corrected text from the text description; The corrected text is replaced with the text to be corrected to obtain a target text.

8. An image-text conversion and correction device, characterized in that: include: An acquisition module, used for acquiring an original image to be processed and inputting the original image into a preset image-to-text model to generate a text description; A simulation module, used for obtaining a simulated image based on the original image, the text description and a preset text image model; A slicing module, used for performing slicing analysis on the original image and the simulated image to obtain image slice pairs and similarity data of each image slice pair; An analysis module, configured to analyze the image slice pair according to the similarity data to generate a corrected text; A correction module is used to correct the text description by using the correction text to obtain a target text.

9. A device for converting and correcting images and texts, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

Images