Picture object processing method and performance determination method and device of picture processing model

Abstract

The invention relates to the technical field of computer processing, and discloses a picture object processing method and a picture processing model performance determination method and device.The picture object processing method comprises the steps that a target file is acquired to obtain a first picture containing a target object in the target file; and inputting the first picture into a target picture processing model to obtain a first text corresponding to the first picture, the target picture processing model being obtained by performing joint training on an initial picture processing model and an initial picture discrimination model, and the initial picture discrimination model being used for evaluating the picture processing performance of the initial picture processing model. The image-text conversion processing is carried out on the target image processing model obtained after joint training is carried out based on the initial image processing model and the initial image discrimination model, so that the understanding and analysis capabilities of the target image processing model can be effectively enhanced, the recognition quality of the target object can be ensured, and the recognition accuracy is effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer processing technology, and in particular to a method for processing an image object, and a method and device for determining the performance of an image processing model. Background Art

[0002] In the related art, a target element can be identified from a file containing multiple elements by image recognition, but a method is urgently needed to improve the recognition accuracy of the target element. Summary of the invention

[0003] In view of this, the present disclosure provides a method for processing an image object, a method and a device for determining the performance of an image processing model, so as to solve the problem of low accuracy in converting images into text.

[0004] In a first aspect, the present disclosure provides a method for processing an image object, comprising:

[0005] Acquire a target file to obtain a first picture containing a target object in the target file;

[0006] The first image is input into a target image processing model to obtain a first text corresponding to the first image. The target image processing model is obtained by jointly training an initial image processing model and an initial image discrimination model. The initial image discrimination model is used to evaluate the image processing performance of the initial image processing model.

[0007] In a second aspect, the present disclosure provides a method for determining the performance of an image processing model, the method comprising:

[0008] Acquire at least one first image processing model and a second image corresponding to the first image processing model, wherein the second image includes a target object;

[0009] The second image is input into the corresponding first image processing model to obtain a processed text, and the target image discrimination model is used to determine the similarity between the features of the second image and the features of the rendered image corresponding to the processed text; the target image discrimination model is obtained by jointly training the initial image processing model and the initial image discrimination model, and the initial image discrimination model is used to evaluate the image processing performance of the initial image processing model;

[0010] The processing performance of the first image processing model is determined based on the similarity.

[0011] In a third aspect, the present disclosure provides a picture object processing device, the device comprising:

[0012] A first acquisition module is used to acquire a target file to obtain a first picture containing a target object in the target file;

[0013] The recognition module is used to input the first image into the target image processing model to obtain the first text corresponding to the first image. The target image processing model is obtained by jointly training the initial image processing model and the initial image discrimination model. The initial image discrimination model is used to evaluate the image processing performance of the initial image processing model.

[0014] In a fourth aspect, the present disclosure provides a device for determining performance of an image processing model, the device comprising:

[0015] A second acquisition module, used to acquire at least one first image processing model and a second image corresponding to the first image processing model, wherein the second image includes a target object;

[0016] a first processing module, configured to input the second image into the corresponding first image processing model to obtain a processed text, and to use a target image discrimination model to determine the similarity between features of the second image and features of a rendered image corresponding to the processed text; the target image discrimination model is obtained by jointly training an initial image processing model and an initial image discrimination model, and the initial image discrimination model is used to evaluate the image processing performance of the initial image processing model;

[0017] The second processing module is used to determine the processing performance of the first image processing model based on the similarity.

[0018] In a fifth aspect, the present disclosure provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the image object processing method of the above-mentioned first aspect or any corresponding embodiment or the performance determination method of the image processing model of the above-mentioned second aspect or any corresponding embodiment.

[0019] In a sixth aspect, the present disclosure provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image object processing method of the above-mentioned first aspect or any corresponding embodiment thereof, or the performance determination method of the image processing model of the above-mentioned second aspect or any corresponding embodiment thereof.

[0020] In the seventh aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the image object processing method of the above-mentioned first aspect or any corresponding embodiment or the performance determination method of the image processing model of the above-mentioned second aspect or any corresponding embodiment.

[0021] The image object processing method provided in this embodiment performs image-to-text conversion processing based on the target image processing model obtained after joint training of the initial image processing model and the initial image discrimination model. This can effectively enhance the understanding and analysis capabilities of the target image processing model, thereby ensuring the recognition quality of the target object and effectively improving the recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the specific embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the specific embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1 is a flowchart of a method for processing a picture object provided according to an embodiment of the present disclosure;

[0024] Figure 2 is a schematic diagram of a model structure provided according to an embodiment of the present disclosure;

[0025] Figure 3 is a flowchart of a joint training method provided according to an embodiment of the present disclosure;

[0026] Figure 4 is a flowchart of a method for determining the performance of an image processing model provided in an embodiment of the present disclosure;

[0027] Figure 5 is a structural block diagram of a picture object processing device provided according to an embodiment of the present disclosure;

[0028] Figure 6 is a structural block diagram of a device for determining performance of an image processing model according to an embodiment of the present disclosure;

[0029] Figure 7 It is a schematic diagram of the hardware structure of the electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0031] In related technologies, target elements can be identified from a file containing multiple elements by image recognition. However, in actual applications, the file's built-in parser may not be able to effectively understand and process these target elements, resulting in poor overall parsing quality and affecting the recognition accuracy of the target elements.

[0032] In view of this, an embodiment of the present disclosure provides a method for processing image objects, which can effectively improve the recognition accuracy of image objects.

[0033] According to an embodiment of the present disclosure, an embodiment of a method for processing a picture object is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] In this embodiment, a method for processing an image object is provided, which can be used in the above-mentioned electronic devices, such as notebooks, computers, etc. Figure 1 is a flowchart of a method for processing a picture object according to an embodiment of the present disclosure. Figure 1 As shown, the process includes the following steps:

[0035] Step S101, acquiring a target file to obtain a first picture containing a target object in the target file.

[0036] The target file refers to the file that includes the target object, and the source can be local or cloud. The file type of the target file can include image files or document files, which can be determined according to actual needs. Among them, the file format of the image file can be JPEG, PNG, or GIF. The file format of the document file can be PDF file or Excel file. The element type of the target object can be a specified type, such as: formula element, graphic element, etc.

[0037] By acquiring the target file, the first image containing the target object to be processed can be clearly identified, which helps to improve the pertinence and effectiveness of the subsequent recognition processing.

[0038] Step S102: input the first image into the target image processing model to obtain the first text corresponding to the first image.

[0039] Among them, the target image processing model is obtained by jointly training the initial image processing model and the initial image discrimination model, and the initial image discrimination model is used to evaluate the image processing performance of the initial image processing model. The target image processing model can be understood as a multimodal model used to transform images to convert input images into text. The initial image processing model is the basic model used by the target image processing model during the training process. Jointly training the initial image processing model and the initial image discrimination model can continuously improve and optimize the processing capabilities of the initial image processing model during the training process, combined with the evaluation results of the initial image discrimination model, and improve its own performance, so that the performance of the final target image processing model is better, which is more conducive to improving the quality and reliability of image-to-text conversion.

[0040] Therefore, after obtaining the first image, the first image is input into the target image processing model obtained after joint training for processing and analysis, which can ensure the generation quality of the first text and thus effectively improve the recognition accuracy of the image object.

[0041] The image object processing method provided in this embodiment performs image-to-text conversion processing based on the target image processing model obtained after joint training of the initial image processing model and the initial image discrimination model. This can effectively enhance the understanding and analysis capabilities of the target image processing model, thereby ensuring the recognition quality of the target object and effectively improving the recognition accuracy.

[0042] In some optional embodiments, the target object is a formula element, and the format of the first text is a specified text format. For example, the formula element is a mathematical formula or a chemical formula. If the formula element is a mathematical formula, the format of the first text can be LaTex or Markdown format. If the formula element is a chemical formula, the format of the first text can be ChemDraw format or SMILES format, etc.

[0043] In some optional embodiments, such as Figure 2 As shown, the initial image processing model and the initial image discrimination model respectively include a first visual encoder and a second visual encoder with the same structure; the initial image discrimination model also includes a discriminator and a renderer, the renderer is used to render the predicted text output by the initial image processing model into a predicted image, the second visual encoder is used to encode the predicted image to obtain a second feature, the discriminator is used to respectively process the first feature and the second feature output by the first visual encoder to obtain a discriminant loss, and the discriminant loss is used to jointly train the initial image processing model and the initial image discrimination model.

[0044] In order to facilitate joint training and ensure that the initial image processing model and the initial image discrimination model have their corresponding valid inputs, the initial image processing model is connected to the initial image discrimination model, so that in the subsequent joint training process, the initial image processing model and the initial image discrimination model can cooperate with each other, thereby achieving the purpose of improving the performance of image processing and text generation.

[0045] In an optional implementation scenario, the initial image processing model further includes a decoder, and the connection relationship between the initial image processing model and the initial image discrimination model can be as follows: Figure 2 As shown. Among them, the first visual encoder is respectively connected to the decoder and the discriminator of the initial image discrimination model, the decoder is connected to the renderer, the renderer is connected to the second visual encoder, and the second visual encoder is connected to the discriminator. That is, the input of the discriminator is the first feature output by the first visual decoder and the second feature output by the second visual encoder. The input of the decoder is the first feature, and the output is the predicted text. The renderer is used to render the predicted text output by the decoder into a predicted picture, and input it into the second visual decoder to obtain the second feature. Through this connection method for joint training, the initial image processing model can learn how to generate more accurate predicted text, thereby improving the quality of image-to-text conversion, so that the generated text can more accurately describe the content of the picture, while also improving the generalization ability and robustness of the model.

[0046] Combination Figure 2 The joint training structure diagram shown in FIG. 1 is a schematic diagram of a joint training structure. In this embodiment, a joint training method is also provided, which can be used for the above-mentioned electronic devices, such as notebooks, computers, etc. Figure 3 is a flow chart of a joint training method according to an embodiment of the present disclosure. Figure 3 As shown, the process includes the following steps:

[0047] Step S301, obtaining a sample image containing a target object and a text label corresponding to the sample image.

[0048] In order to make the model training more targeted, sample images containing target object images are obtained in advance. The source of the samples may include the Internet, databases, etc.

[0049] An accurate text label is configured for each sample image so that the initial image processing model can accurately describe the content of the image through the label during subsequent training, thus laying the foundation for subsequent model training.

[0050] In some optional implementation scenarios, in order to reduce the difficulty of training, the obtained sample images are converted according to a specified format to ensure that the size dimensions of each sample image are the same during subsequent training.

[0051] In other optional implementation scenarios, the pre-acquired images can be screened and sorted to remove poor quality or irrelevant images to reduce noise and interference information and ensure the cleanliness of the sample images. The subsequent initial image processing model can then focus more on learning features and patterns related to the target object during training, which is beneficial to improving training efficiency and model accuracy.

[0052] Step S302: input the sample image into the initial image processing model to obtain predicted text.

[0053] The sample image is sent to the initial image processing model so that the initial image processing model can perform targeted analysis and understanding of the sample image based on the current model parameters and obtain the corresponding predicted text, which can provide a basis for subsequent loss calculation and model optimization. At the same time, it also enables the initial image processing model to continuously learn and improve its understanding of images and text generation capabilities.

[0054] Step S303: obtaining text loss based on the difference between the predicted text and the text label.

[0055] The predicted text and text label are input into the loss function, the difference between the predicted text and the actual text label is compared, and the text loss is calculated based on this difference, so as to reflect the accuracy and performance of the initial image processing model in generating text through the text loss. In some optional application scenarios, the loss function for calculating the text loss is The corresponding expression can be as follows:

[0056] Among them, y represents the current sample image, represents the predicted text corresponding to the current sample image, C represents the total number of sample images, and c represents the number of tokens corresponding to the text label of the current sample image.

[0057] Step S304: input the predicted text into the initial image discrimination model to obtain the discrimination loss.

[0058] The predicted text generated by the initial image processing model is input into the initial image discrimination model. The initial image discrimination model can be used to compare the similarity between the predicted text and the text label, and then the discrimination loss corresponding to the initial image discrimination model can be obtained to help evaluate the initial image discrimination model's ability to judge the authenticity of the generated text, and then perform targeted optimization to improve its judgment accuracy on the authenticity of the text and enhance the reliability and security of the overall training architecture.

[0059] In some optional implementations, the above step S304 includes:

[0060] Step a1, using a renderer to render the predicted text to obtain a predicted image;

[0061] Step a2, using a second visual encoder to encode the predicted image to obtain a second feature;

[0062] Step a3: Use the discriminator to calculate the loss of the first feature and the second feature output by the first visual encoder to obtain the discriminant loss.

[0063] In order to make it easier for the discriminator to clearly distinguish the difference between the predicted text and the sample image, a renderer is used to convert the predicted text output by the initial image processing model into a predicted image, and a second visual encoder with the same structure as the first visual encoder is used to encode the predicted image to obtain a second feature, so as to represent the key information of the predicted image through the second feature, which is convenient for subsequent analysis and comparison.

[0064] The first feature and the second feature corresponding to the sample image output by the first visual encoder are input into the discriminator to compare and analyze the difference between the two by feature matching, and to calculate the loss to obtain the discriminant loss. The purpose of the discriminant loss is to measure the difference between the predicted image and the sample image in the feature space, so as to judge the performance of the initial image processing model and the quality of the generated predicted text.

[0065] In some optional application scenarios, the loss function l for calculating the discriminant loss is dis The corresponding expression can be as follows:

[0066] Among them, z represents the first feature, represents the corresponding second feature, and D represents the discriminator.

[0067] Step S305 , based on the fusion result of the text loss and the discrimination loss, the initial image processing model and the initial image discrimination model are jointly trained to obtain a target image processing model and a target image discrimination model.

[0068] The text loss and the discrimination loss are fused, and the parameters of the initial image processing model and the initial image discrimination model are adjusted simultaneously according to the fusion results for joint training. Through continuous optimization, a target image processing model and a target image discrimination model with better performance are finally obtained. For example, the gradient of the overall model parameters can be calculated according to the fusion results by back propagation, and the parameters of the initial image processing model and the initial image discrimination model can be updated and optimized according to the gradient information. By repeating the above steps and performing multiple iterative training until the performance of the model reaches the expected goal, the target image processing model can generate more accurate text descriptions, and the target image discrimination model can more accurately judge the authenticity of the text, thereby improving the overall processing performance and reliability.

[0069] In some optional implementations, the above step S305 includes:

[0070] Step b1, respectively obtaining the first coefficient and the second coefficient corresponding to the text loss and the discrimination loss;

[0071] Step b2, performing weighted calculation based on the first coefficient and the second coefficient, and the corresponding text loss and discrimination loss to obtain a fusion loss;

[0072] Step b3: jointly train the initial image processing model and the initial image discrimination model based on the fusion loss to obtain the target image processing model and the target image discrimination model.

[0073] Specifically, determine the coefficients corresponding to the text loss and the discriminant loss, that is, the first coefficient λ 1 and the second coefficient λ 2 The first coefficient and the second coefficient can be set according to actual conditions, for example, according to the characteristics of the data, the performance requirements of the model or determined through experiments. The sum of the first coefficient and the second coefficient is 1.

[0074] The text loss is multiplied by the first coefficient, the discrimination loss is multiplied by the second coefficient, and then the two results are added together. The fusion loss obtained can comprehensively consider the information of the text loss and the discrimination loss to reflect the overall performance of the initial image processing model and the initial image discrimination model according to the obtained value. For example: The formula for determining the fusion loss is as follows:

[0075] The initial image processing model and the initial image discrimination model are jointly trained using the obtained fusion loss. During the training process, the fusion loss can be minimized by adjusting the model parameters. In addition, as the training progresses, the model will gradually learn how to better process images and generate text, as well as how to make more accurate discriminations. When the fusion loss reaches a better value, the training ends, and the target image processing model and the target image discrimination model are obtained.

[0076] In general, by fusing the text loss and the discriminant loss, and using the fused loss to optimize the initial image processing model and the initial image discriminant model, the performance and accuracy of the model can be effectively improved.

[0077] The joint training method provided in this embodiment can combine the text generation ability of the image processing model and the evaluation ability of the discriminant model during the training process, and then through the joint effect of text loss and discriminant loss, the target image processing model can better learn the relationship between the features of the image and the corresponding text description, thereby improving the accuracy and reliability of image processing and text generation.

[0078] As one or more specific application embodiments of the embodiments of the present disclosure, the above-mentioned image object processing method can be applied to the application scenario of performing image-to-text conversion processing on mathematical formulas in files in Portable Document Format (PDF) format, so as to provide a powerful and reliable data foundation for the downstream large language model.

[0079] Based on the same inventive concept, the present invention also provides a method for determining the performance of an image processing model, which can be used in the above-mentioned electronic devices, such as notebooks, computers, etc. Figure 4 is a flowchart of a method for determining the performance of an image processing model according to an embodiment of the present disclosure. Figure 4 As shown, the process includes the following steps:

[0080] Step S401, obtaining at least one first image processing model and a second image corresponding to the first image processing model.

[0081] The second image includes the target object. The first image processing model is a model for converting the image including the target object into a corresponding description text. The first image processing model can be obtained based on the joint training method described above, or can be obtained through other training methods.

[0082] Since the image processing capability of the first image processing model is unknown, in order to determine the processing capability of the first image processing model, a second image for testing the first image processing model is obtained.

[0083] Step S402, input the second image into the corresponding first image processing model to obtain a processed text, and use the target image discrimination model to determine the similarity between the features of the second image and the features of the rendered image corresponding to the processed text.

[0084] The target image discrimination model is obtained by jointly training the initial image processing model and the initial image discrimination model, and the initial image discrimination model is used to evaluate the image processing performance of the initial image processing model. That is, the initial image processing model and the initial image discrimination model are jointly trained in advance, and then in response to the end of the training, the target image processing model is determined based on the current model parameters of the initial image processing model; and the target image discrimination model is determined based on the current model parameters of the initial image discrimination model.

[0085] The processed text can be understood as the processing result obtained by the first image processing model after performing image-text conversion processing on the second image, and the processed text is used to describe the second image. Since the model performance can be analyzed based on the output result thereof, in order to improve the reliability of the analysis result, the obtained processed text is rendered by the renderer in the target image discrimination model to obtain the corresponding rendered image. The features of the second image and the rendered image are extracted respectively, and then the discriminator in the target image discrimination model is used for matching to judge the similarity between the features of the image and the features of the corresponding processed text rendered image, so that the obtained similarity can be used to evaluate the performance of the first image processing model later. For example, if the similarity is high, it means that the first image processing model can better understand the image content and generate an accurate text description; on the contrary, if the similarity is low, the first image processing model may need to be further improved or optimized to improve the performance and accuracy so that it can better meet the needs of practical applications. Among them, since the features of the second image are obtained by encoding processing by the first visual encoder in the target image processing model, the features of the rendered image are obtained by encoding processing by the second visual encoder in the target image discrimination model. In some optional examples, to save costs, the first visual encoder and the second visual encoder may be the same visual encoder.

[0086] Step S403: determining the processing performance of the first image processing model based on the similarity.

[0087] If the similarity is high, it means that the first image processing model can better convert the information in the second image into accurate processed text, and the rendered image features are close to the original image features, indicating that the processing performance of the model is good.

[0088] If the similarity is low, it means that the first image processing model may have deficiencies in image processing and text generation, or there is a large deviation when rendering the processed text back to the image, reflecting that the processing performance of the model needs to be improved.

[0089] In some examples, the discriminator is a binary classification discriminator, and the output result is a probability value. Among them, the larger the probability value, the higher the representation similarity, and the smaller the probability value, the lower the representation similarity. The performance determination method of the image processing model provided in this embodiment determines the processing performance of the first image processing model based on the similarity, which can intuitively understand the performance of the first image processing model, and provides a basis for further improving and optimizing the model, thereby helping to improve the quality and effect of the first image processing model, so that it can better meet the needs of practical applications.

[0090] In some optional implementations, when the number of acquired first image processing models is multiple, and the second images corresponding to all first image processing models are the same, the performance determination method of the above-mentioned image processing model further includes:

[0091] Step c1, comparing the similarities corresponding to the first image processing model;

[0092] Step c2: based on the similarity comparison result, all first image processing models are screened to determine a second image processing model for processing the second image.

[0093] Specifically, since the second images corresponding to each first image processing model are the same, but the training process is not exactly the same, in order to determine the image processing capabilities of each first image processing model, the similarities corresponding to each two first image processing models are compared, and the comparison results are determined, and then one or more first image processing models with higher similarity are selected as the second image processing model, so that it can better meet the needs of actual applications. Among them, the second image processing model can be determined according to actual needs and application scenarios.

[0094] By selecting the second image processing model in the above manner, the selection criteria can be made more universal and objective, so that the obtained selection results are more scientific and reliable. At the same time, this also helps to optimize resource allocation and improve work efficiency.

[0095] As one or more specific application embodiments of the embodiments of the present disclosure, taking the target file as a file in Portable Document Format (PDF) format and the target object as a mathematical formula as an example, the process of image object processing can be as follows: obtain the target file containing the mathematical formula, and input the first image corresponding to the target file into the target image processing model to perform image-to-text conversion processing, and then obtain the first text corresponding to the first image.

[0096] Furthermore, in order to evaluate the image processing capability of the target image processing model, the first text and the first image feature are input into the target image discrimination model. The first image feature is obtained by encoding the first image from the first visual encoder in the target image processing model. The first text is rendered by the renderer in the target image discrimination model to obtain a rendered image, and the rendered image is encoded by the second visual encoder in the target image discrimination model to obtain a second image feature. The first image feature and the second image feature are input into the discriminator in the target image discrimination model to determine the similarity between the two, and then the similarity is used to evaluate the processing performance of the target image processing model.

[0097] Preferably, the target image processing model and the target image discrimination model are obtained by joint training.

[0098] In this embodiment, a picture object processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0099] This embodiment provides a picture object processing device, such as Figure 5 As shown, including:

[0100] A first acquisition module 501 is used to acquire a target file to obtain a first picture containing a target object in the target file;

[0101] The recognition module 502 is used to input the first image into the target image processing model to obtain the first text corresponding to the first image. The target image processing model is obtained by jointly training the initial image processing model and the initial image discrimination model. The initial image discrimination model is used to evaluate the image processing performance of the initial image processing model.

[0102] In some optional implementations, the initial image processing model and the initial image discrimination model respectively include a first visual encoder and a second visual encoder having the same structure;

[0103] The initial image discrimination model also includes a discriminator and a renderer. The renderer is used to render the predicted text output by the initial image processing model into a predicted image. The second visual encoder is used to encode the predicted image to obtain a second feature. The discriminator is used to process the first feature and the second feature output by the first visual encoder respectively to obtain a discrimination loss. The discrimination loss is used to jointly train the initial image processing model and the initial image discrimination model.

[0104] In some optional embodiments, the joint training device includes:

[0105] A sample acquisition module is used to obtain sample images containing target objects and text labels corresponding to the sample images;

[0106] The prediction module is used to input the sample image into the initial image processing model to obtain the predicted text;

[0107] A first execution module, configured to obtain a text loss based on a difference between the predicted text and the text label;

[0108] The second execution module is used to input the predicted text into the initial image discrimination model to obtain the discrimination loss;

[0109] The training module is used to jointly train the initial image processing model and the initial image discrimination model based on the fusion result of the text loss and the discrimination loss to obtain the target image processing model and the target image discrimination model.

[0110] In some optional implementations, the second execution module includes:

[0111] A rendering unit, used for rendering the predicted text using a renderer to obtain a predicted image;

[0112] An encoding unit, configured to encode the predicted image using a second visual encoder to obtain a second feature;

[0113] The loss determination unit is used to use the discriminator to perform loss calculation on the first feature and the second feature output by the first visual encoder to obtain a discriminant loss.

[0114] In some optional embodiments, the training module includes:

[0115] A third execution unit, used to obtain a first coefficient and a second coefficient corresponding to the text loss and the discrimination loss respectively;

[0116] A fourth execution unit, configured to perform weighted calculation based on the first coefficient and the second coefficient, and the corresponding text loss and discrimination loss to obtain a fusion loss;

[0117] The fifth execution unit is used to jointly train the initial image processing model and the initial image discrimination model based on the fusion loss to obtain the target image processing model and the target image discrimination model.

[0118] In some optional implementations, the target object is a formula element, and the format of the first text is a specified text format.

[0119] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0120] The image object processing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0121] In this embodiment, a performance determination device for an image processing model is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0122] This embodiment provides a device for determining the performance of an image processing model, such as Figure 6 As shown, including:

[0123] A second acquisition module 601 is used to acquire at least one first image processing model and a second image corresponding to the first image processing model, wherein the second image includes a target object;

[0124] The first processing module 602 is used to input the second image into the corresponding first image processing model to obtain a processed text, and use the target image discrimination model to determine the similarity between the features of the second image and the features of the rendered image corresponding to the processed text; the target image discrimination model is obtained by jointly training the initial image processing model and the initial image discrimination model, and the initial image discrimination model is used to evaluate the image processing performance of the initial image processing model;

[0125] The second processing module 603 is used to determine the processing performance of the first image processing model based on the similarity.

[0126] In some optional implementations, the number of acquired first image processing models is multiple, and the second images corresponding to all first image processing models are the same, and the device further includes:

[0127] A comparison module, used for comparing the similarities corresponding to the first image processing model;

[0128] The screening module is used to screen all first image processing models based on the size comparison result of the similarity, and determine the second image processing model used to process the second image.

[0129] The performance determination device of the image processing model in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above-mentioned functions.

[0130] The present disclosure also provides an electronic device having the above Figure 5 The image object processing device shown and / or the above Figure 6 The performance of the picture processing model shown is determined by the device.

[0131] See also Figure 7 , Figure 7 is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present disclosure, such as Figure 7 As shown, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0132] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0133] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0134] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0135] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0136] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0137] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0138] The embodiments of the present disclosure also provide a computer-readable storage medium. The above-mentioned method according to the embodiments of the present disclosure can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium and downloaded through a network, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0139] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0140] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0141] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0142] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0143] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0144] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for processing a picture object, characterized in that: The method comprises: Acquire a target file to obtain a first picture containing a target object in the target file; The first image is input into a target image processing model to obtain a first text corresponding to the first image. The target image processing model is obtained by jointly training an initial image processing model and an initial image discrimination model. The initial image discrimination model is used to evaluate the image processing performance of the initial image processing model.

2. The method according to claim 1, characterized in that The initial image processing model and the initial image discrimination model respectively include a first visual encoder and a second visual encoder with the same structure; The initial image discrimination model also includes a discriminator and a renderer. The renderer is used to render the predicted text output by the initial image processing model into a predicted image. The second visual encoder is used to encode the predicted image to obtain a second feature. The discriminator is used to process the first feature and the second feature output by the first visual encoder respectively to obtain a discrimination loss. The discrimination loss is used to jointly train the initial image processing model and the initial image discrimination model.

3. The method according to claim 2, characterized in that The joint training method includes: Obtain a sample image containing the target object and a text label corresponding to the sample image; Inputting the sample image into the initial image processing model to obtain the predicted text; Obtaining a text loss based on a difference between the predicted text and the text label; Inputting the predicted text into the initial image discrimination model to obtain the discrimination loss; Based on the fusion result of the text loss and the discrimination loss, the initial image processing model and the initial image discrimination model are jointly trained to obtain the target image processing model and the target image discrimination model.

4. The method according to claim 3, characterized in that The step of inputting the predicted text into the initial image discrimination model to obtain the discrimination loss includes: Rendering the predicted text using the renderer to obtain the predicted image; Encoding the predicted picture using the second visual encoder to obtain a second feature; The discriminator is used to perform loss calculation on the first feature and the second feature output by the first visual encoder to obtain the discriminant loss.

5. The method according to claim 3, characterized in that: The initial image processing model and the initial image discrimination model are jointly trained based on the fusion result of the text loss and the discrimination loss to obtain the target image processing model and the target image discrimination model, including: Obtaining a first coefficient and a second coefficient corresponding to the text loss and the discrimination loss respectively; Based on the first coefficient and the second coefficient, weighted calculation is performed with the corresponding text loss and the discrimination loss to obtain a fusion loss; The initial image processing model and the initial image discrimination model are jointly trained based on the fusion loss to obtain the target image processing model and the target image discrimination model.

6. The method according to any one of claims 1 to 5, characterized in that The target object is a formula element, and the format of the first text is a specified text format.

7. A method for determining the performance of an image processing model, characterized in that: The method comprises: Acquire at least one first image processing model and a second image corresponding to the first image processing model, wherein the second image includes a target object; Inputting the second image into the corresponding first image processing model to obtain a processed text, and using a target image discrimination model to determine the similarity between features of the second image and features of a rendered image corresponding to the processed text; the target image discrimination model is obtained by jointly training an initial image processing model and an initial image discrimination model, and the initial image discrimination model is used to evaluate the image processing performance of the initial image processing model; The processing performance of the first image processing model is determined based on the similarity.

8. The method according to claim 7, characterized in that The number of acquired first image processing models is multiple, and the second images corresponding to all the first image processing models are the same, and the method further includes: Comparing the similarities corresponding to the first image processing models; Based on the comparison result of the similarity, all the first image processing models are screened to determine a second image processing model for processing the second image.

9. A picture object processing device, characterized in that: The device comprises: A first acquisition module is used to acquire a target file to obtain a first picture containing a target object in the target file; The recognition module is used to input the first image into a target image processing model to obtain a first text corresponding to the first image. The target image processing model is obtained by jointly training an initial image processing model and an initial image discrimination model. The initial image discrimination model is used to evaluate the image processing performance of the initial image processing model.

10. A device for determining the performance of an image processing model, characterized in that: The device comprises: A second acquisition module, used to acquire at least one first image processing model and a second image corresponding to the first image processing model, wherein the second image includes a target object; a first processing module, configured to input the second image into a corresponding first image processing model to obtain a processed text, and to determine the similarity between features of the second image and features of a rendered image corresponding to the processed text using a target image discrimination model; the target image discrimination model is obtained by jointly training an initial image processing model and an initial image discrimination model, and the initial image discrimination model is used to evaluate the image processing performance of the initial image processing model; The second processing module is used to determine the processing performance of the first image processing model based on the similarity.

11. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the image object processing described in any one of claims 1 to 6 or the performance determination method of the image processing model described in any one of claims 7 to 8 by executing the computer instructions.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image object processing described in any one of claims 1 to 6 or the performance determination method of the image processing model described in any one of claims 7 to 8.

13. A computer program product, characterized in that It comprises computer instructions, wherein the computer instructions are used to cause a computer to execute the picture object processing described in any one of claims 1 to 6 or the performance determination method of the picture processing model described in any one of claims 7 to 8.

Images