Image description data processing method and device, equipment and medium

Abstract

The invention provides an image description data processing method and device, equipment and a medium, and relates to the field of image processing, in particular to the technical field of artificial intelligence and large models. According to the specific implementation scheme, a sample image is input into a to-be-detected target model to be processed, and description content output by the target model is obtained; determining at least one target quality evaluation mode according to the sample image and the description content; and determining a quality evaluation result of the target model for the sample image according to the sample image and the description content by adopting each target quality evaluation mode. According to the embodiment of the invention, the image description content quality output by the model can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing, specifically to the field of artificial intelligence and large model technology, and in particular to an image description data processing method, device, equipment and medium. Background Art

[0002] Multimodal Large Language Models (MLLMs) are large language models that fuse different media data (such as text, images, audio, and video).

[0003] With the rapid development of large multimodal models, efficiently and automatically evaluating the generation quality and performance of these models in practical applications is currently a key issue. Summary of the invention

[0004] The present disclosure provides an image description data processing method, device, equipment and medium.

[0005] According to one aspect of the present disclosure, a method for processing image description data is provided, comprising:

[0006] Inputting the sample image into the target model to be detected for processing, and obtaining the description content output by the target model;

[0007] Determining at least one target quality assessment method according to the sample image and the description content;

[0008] Each of the target quality assessment methods is adopted to determine the quality assessment result of the target model for the sample image according to the sample image and the description content.

[0009] According to one aspect of the present disclosure, there is provided an image description data processing device, comprising:

[0010] A description content acquisition module is used to input the sample image into the target model to be detected for processing, and obtain the description content output by the target model;

[0011] A quality assessment method determination module, used to determine at least one target quality assessment method according to the sample image and the description content;

[0012] The quality assessment result determination module is used to determine the quality assessment result of the target model for the sample image according to the sample image and the description content by adopting each of the target quality assessment methods.

[0013] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the image description data processing method described in any embodiment of the present disclosure.

[0017] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the image description data processing method described in any embodiment of the present disclosure.

[0018] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the image description data processing method described in any embodiment of the present disclosure is implemented.

[0019] The disclosed embodiments can improve the quality of image description content output by the model.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a flowchart of an image description data processing method disclosed in an embodiment of the present disclosure;

[0023] Figure 2 is a flowchart of another method for processing image description data according to an embodiment of the present disclosure;

[0024] Figure 3 is a flowchart of another method for processing image description data according to an embodiment of the present disclosure;

[0025] Figure 4 is a scene graph of an image description data processing method disclosed in an embodiment of the present disclosure;

[0026] Figure 5 is a schematic diagram of the structure of an image description data processing device disclosed in an embodiment of the present disclosure;

[0027] Figure 6 The invention is a block diagram of an electronic device according to the method for processing image description data disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] Figure 1 This is a flowchart of an image description data processing method disclosed in an embodiment of the present disclosure. This embodiment can be applied to the case where the quality of the model output result of the image description is determined. The method of this embodiment can be executed by an image description data processing device, which can be implemented in software and / or hardware, and is specifically configured in an electronic device with certain data computing capabilities, which can be a server device.

[0030] S101: Input a sample image into a target model to be detected for processing, and obtain a description content output by the target model.

[0031] The target model is used to describe the image and obtain natural language content. The media type of the natural language content may be text, voice, image or video, etc. Exemplarily, the target model is used to input a sample image and output text. The description content may refer to the content that describes the image content. The description content may include the content of the foreground and / or background of the sample image, and the description content may also include the content included in the sample image and the content outside the sample image that is extended based on the included content but is related to the included content. The object of quality detection is the target model. The target model may be a large language model, and more specifically, the target model is a multimodal large language model. Exemplarily, the target model may encode the sample image to obtain image features, decode the image features to obtain description content; or the target model may also input interactive information, encode the interactive information to obtain text features, fuse the text features and image features, decode the fused features to obtain description content.

[0032] S102: Determine at least one target quality assessment method according to the sample image and the description content.

[0033] At least one alternative quality assessment method can be pre-set, and at least one alternative quality assessment method can be screened and determined as a target quality assessment method. The sample image and the description content describe information such as the field to be identified and the content to be identified, and the target quality assessment method suitable for the field to which the sample image and the description content belong can be screened out from the alternative quality assessment methods based on the identifiable information.

[0034] Key information can be identified from the description content and sample images, and the target quality assessment method can be determined based on the key information. For example, keywords or text features can be identified from the description content; image features can be identified from the sample images. The target domain is determined based on keywords (or text features) and image features, and a target quality assessment method adapted to the target domain is selected. Among them, the image features can include at least one of the following: vector features, image texture, color, contrast, image type, etc.

[0035] An adapted field can be set for each alternative quality assessment method, an adapted field can be set in the specific assessment rules involved in each alternative quality assessment method, an adapted field can be set in the knowledge base involved in each alternative quality assessment method, and an adapted field can be set in the model involved in each quality assessment method. The alternative quality assessment method corresponding to the target field can be selected, and the assessment rules, knowledge base or model corresponding to the target field can also be selected.

[0036] Alternatively, a large language model can be used to directly select a target quality assessment method from alternative quality assessment methods based on sample images and description content. A target quality assessment method corresponding to an image and corresponding description content is generated in advance as an assessment method sample. The large language model is supervised based on a large number of assessment method samples, and the trained large language model is used to determine the target quality assessment method based on the sample image and description content. The large language models in the embodiments of the present disclosure are all large language models trained in general fields, and on the basis of such large language models, the knowledge of the target field is again used for supervised training to obtain a quality detection model.

[0037] S103: using each of the target quality assessment methods, and determining a quality assessment result of the target model for the sample image according to the sample image and the description content.

[0038] When there are multiple target quality assessment methods, each target quality assessment method may be selected to determine the assessment result of the target quality assessment method according to the content of the target quality assessment method, and the assessment results of multiple target quality assessment methods may be fused to obtain the quality assessment result of the target model. The fusion method may be accumulation or weighted summation. Alternatively, multiple target assessment methods may be combined according to the content of the target quality assessment method to directly determine the quality assessment result of the target model.

[0039] The quality assessment result of the target model can be used to indicate whether the target model is released online. For example, when the quality assessment result of the target model is greater than or equal to the preset quality threshold, it is determined to be released online; when the quality assessment result of the target model is less than the preset quality threshold, it is determined to continue iterative optimization. The quality assessment results of the target model can include multiple versions of quality assessment results. According to the quality assessment results of multiple versions of the target model, the optimal version can be determined, so as to select a version of the target model with the best quality. Among them, the best quality can mean the highest quality assessment result. It is also possible to obtain the quality assessment results of multiple target models with different structures, compare the quality assessment results of each target model, and select the target model with the best quality.

[0040] According to the technical solution disclosed in the present invention, by determining at least one target quality assessment method based on sample images and description content, a quality assessment method adapted to the corresponding image content can be determined to perform quality assessment on the target model. The evaluation targets of different image contents can be adapted to conduct targeted assessments of the description quality of the large model in specific content. The image description quality of the large model can be comprehensively and accurately assessed, which is beneficial for selecting high-quality models and controlling the quality of the online model in model iteration, thereby improving image processing quality.

[0041] In an optional embodiment, after determining the quality assessment result of the target model for the sample image, it also includes: obtaining the quality assessment results of the target model corresponding to multiple other images; and determining the quality assessment information of the target model based on the quality assessment result of the sample image and the quality assessment results corresponding to each of the other images.

[0042] Other images are used for image description quality detection of the target model like the sample image. The sample image may refer to the image currently being detected for image description quality, and the other images may refer to the sample images of the image description quality detection executed in parallel or asynchronously. A data set may be obtained, which includes a sample image and a large number of other images. For each image in the data set, the target model is processed to obtain description content. Based on each image and the corresponding description content, a corresponding quality assessment result may be obtained. The quality assessment result corresponding to each image in the data set includes the quality assessment result of the sample image and the quality assessment results corresponding to each other image. The quality assessment result of the sample image and the quality assessment results corresponding to each other image are fused to obtain the quality assessment information of the target model. The quality assessment information can be used as the quality assessment conclusion of the target model. In fact, the quality assessment result of a single image is unstable. Quality assessment by multiple images can make the assessment result more stable.

[0043] By determining the quality assessment information of the target quality model through the quality assessment results of multiple images, the assessment results can be made more stable and the consistency and reliability of the assessment results can be improved.

[0044] In an optional embodiment, determining at least one target quality assessment method based on the sample image and the description content includes: inputting the sample image and the description content into an assessment method detection model for processing, and outputting at least one target quality assessment method, the assessment method detection model is trained with knowledge of the target domain, and the sample image and the description content correspond to the target domain.

[0045] The evaluation method detection model is used to input sample images and description content, and output the target quality evaluation method. The evaluation method detection model can be a large language model. Among them, the evaluation method detection model can be a large language model obtained through supervised training of the knowledge of the target domain to which the sample image and the description content belong. After training, the evaluation method detection model has the knowledge of the target domain and can obtain the quality evaluation method related to the target domain, so that the detection result of the image description quality is more professional and reliable.

[0046] The evaluation method detection model can be associated with an alternative quality evaluation method. Add a sample image and a description content to a prompt template (prompt) to obtain input data, input the input data into the evaluation method detection model for processing, and output at least one target quality evaluation method. The resource address of the sample image can be added to the prompt template, or feature extraction can be performed on the sample image, and the extracted image features can be added to the prompt template. Alternatively, a sample image, a description content, and at least one alternative quality evaluation method can be added to a prompt template (prompt) to obtain input data, input the input data into the evaluation method detection model for processing, and output at least one target quality evaluation method. The evaluation method detection model can be a target model or a large language model different from the target model.

[0047] By adopting an evaluation method detection model to determine the target quality evaluation method, and the evaluation method detection model is trained with knowledge of the target domain corresponding to the sample image and the description content, the target quality evaluation method can be adapted to the target domain, making the detection result of the image description quality more professional and reliable.

[0048] In an optional embodiment, each of the target quality assessment methods includes: a direct assessment method and / or a defect assessment method, and each of the target quality assessment methods also includes: a relative assessment method.

[0049] Direct evaluation is used to directly output evaluation results from input sample images and description content, so as to quickly obtain evaluation results. Specifically, direct evaluation can refer to taking sample images and description content as input, and the output result is the evaluation result of direct evaluation. Relative evaluation is used to evaluate the relative advantages and disadvantages of the generated results of different models or different versions under the same input, and help identify the performance differences between models or strategies. Relative evaluation can refer to taking sample images and description content as input, respectively inputting them into different models, and obtaining output results, wherein different models can be models with different structures or versions. Defect assessment can be used to detect defects. Defect assessment can refer to detecting whether the description content has defects corresponding to the evaluation rules based on preset evaluation rules, and determining the evaluation results according to the satisfied evaluation rules. Defect assessment can reflect the deficiencies in the description content in detail.

[0050] Among them, relative evaluation is suitable for performance comparison between models of different structures or different versions of models. Direct evaluation is suitable for simple, general and fast quality evaluation scenarios. Defect evaluation is suitable for complex, specific and sophisticated quality evaluation scenarios.

[0051] By configuring the quality assessment method as direct assessment, relative assessment, and defect assessment, it is possible to select suitable quality assessment results from multiple dimensions, to specifically evaluate the image description quality of the model, and to determine the image description quality by combining multiple quality assessment methods, thereby effectively reducing the assessment errors caused by subjective factors and improving the reliability and fairness of the results.

[0052] In an optional embodiment, the target quality assessment method includes: direct assessment; according to each target quality assessment method, performing quality assessment on the target model, including: inputting the sample image and description content into a large model trained with the target domain corpus for processing to obtain an assessment result of the direct assessment.

[0053] In an optional embodiment, the target quality assessment method includes: defect assessment; according to each target quality assessment method, performing quality assessment on the target model, including: adopting at least one assessment rule of the target scene, and determining the assessment rules satisfied by the target model according to the sample image and the description content; determining the quality assessment result of the target model for the sample image according to the satisfied assessment rules and the assessment results corresponding to each assessment rule.

[0054] In an optional embodiment, the target scene is a medical scene; at least one evaluation rule of the target scene is used to determine the evaluation rules satisfied by the target model based on the sample image and the description content, including: for each evaluation rule, obtaining key information corresponding to the evaluation rule from the description content; for each evaluation rule, detecting whether the key information corresponding to the evaluation rule is consistent with the sample image, and obtaining the consistent key information corresponding to the evaluation rule; for each evaluation rule, judging whether the target model satisfies the evaluation rule based on the information ratio between the consistent information corresponding to the evaluation rule and the description content.

[0055] Figure 2 It is a flowchart of another image description data processing method disclosed in an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution and can be combined with the above optional implementations. Using each of the target quality assessment methods, according to the sample image and the description content, the quality assessment result of the target model for the sample image is determined, which is specifically: each of the target quality assessment methods, the sample image and the description content are input into the quality detection model to obtain the quality assessment result of the target model.

[0056] S201: Input a sample image into a target model to be detected for processing, and obtain a description content output by the target model.

[0057] S202: Determine at least one target quality assessment method according to the sample image and the description content.

[0058] S203: Input each of the target quality assessment methods, the sample image and the description content into a quality detection model to obtain a quality assessment result of the target model.

[0059] The quality detection model may be a large language model. Optionally, the quality detection model may be a target model, and the quality detection model may be a large language model different from the target model. The quality detection model may encode the sample image, the description content, and the information of the target quality assessment method respectively, and fuse the encoded features, and decode the fused features to obtain the quality assessment result. The quality detection model may be pre-supervisedly trained for training samples formed by standard images, standard description content, and standard quality assessment results, so that the quality detection model has the ability to conduct quality assessment.

[0060] In one example, the quality detection model is a large language model, and the sample image, description content, and target quality assessment method are added to a prompt template (prompt) to obtain input data, and the input data is input into the quality detection model for processing, and the quality assessment result of the target model is output. The resource address of the sample image can be added to the prompt template, or the sample image can be feature extracted and the extracted image features can be added to the prompt template. The output result can be the quality assessment result of the target model directly, or it can be the assessment result corresponding to each target quality assessment method.

[0061] Among them, the quality detection model can be a large language model obtained through supervised training of knowledge of the target domain to which the sample images and description contents belong. After training, the quality detection model has knowledge of the target domain, thereby making the detection results of the image description quality more professional and reliable.

[0062] According to the technical solution disclosed in the present invention, by using a quality detection model to process the target quality assessment method, sample images and description content, the efficiency of quality detection can be improved, while the subjectivity of manual evaluation can be reduced and the objectivity of quality detection results can be improved.

[0063] In an optional embodiment, the target quality assessment methods, the sample image and the description content are input into a quality detection model to obtain a quality assessment result of the target model, including: using at least one quality detection model to obtain an intermediate result corresponding to each target quality assessment method output by each quality detection model according to each target quality assessment method, the sample image and the description content; determining an assessment result corresponding to each target quality assessment method according to the intermediate result corresponding to each target quality assessment method output by each quality detection model; and determining a quality assessment result of the target model for the sample image according to the assessment result corresponding to each target quality assessment method.

[0064] The intermediate result may refer to the evaluation result corresponding to a single quality evaluation method. In order to avoid the jitter error of a single quality detection model, multiple quality detection models with the same function may be selected to perform image description quality detection on the same sample image and the same description content, and the average of multiple evaluation results may be taken to reduce the jitter error of a single quality detection model.

[0065] A sample image, a description content corresponding to the sample image and a target quality assessment method are input into a quality detection model to obtain an intermediate result corresponding to the target quality assessment method, the quality detection model, the sample image and the description content. Different intermediate results of the same target quality assessment method and the same sample image are fused to obtain the assessment result corresponding to the target quality assessment method for the same sample image. The assessment results corresponding to the same sample image by each target quality assessment method are fused to obtain the quality assessment result of the target model for the sample image. The fusion can be summed and averaged.

[0066] By obtaining the intermediate results of multiple quality detection models, the intermediate results of different quality detection models are first aggregated to obtain the evaluation results corresponding to each target quality evaluation method, and then the evaluation results corresponding to the target quality evaluation method are aggregated to finally obtain the quality evaluation result of the target model for the sample image. This can reduce the jitter error of a single quality detection model and improve the accuracy of the quality evaluation result of the target model.

[0067] In an optional embodiment, based on the intermediate results corresponding to each of the target quality assessment methods output by each of the quality detection models, the assessment results corresponding to each of the target quality assessment methods are determined, including: eliminating abnormal values ​​in the intermediate results corresponding to each of the target quality assessment methods output by each of the quality detection models; for each target quality assessment method, taking the average of the intermediate results corresponding to the remaining target quality assessment methods to obtain the assessment result corresponding to the target quality assessment result.

[0068] An outlier may be a value that differs greatly from other values. Exemplarily, the maximum and minimum values ​​in the intermediate results may be determined as outliers. The number of outliers may be at least one. For the same target quality assessment method, the maximum and minimum values ​​in the intermediate results output by each quality detection model may be determined as outliers. That is, each target quality assessment method may screen out at least one outlier. An outlier is an abnormal intermediate result caused by the instability of the assessment results between multiple quality detection models.

[0069] By proposing outliers in the intermediate results, incorrect evaluation results can be excluded, thereby improving the stability and reliability of the target quality evaluation results.

[0070] In an optional embodiment, each of the target quality assessment methods includes: a direct assessment method and / or a defect assessment method, and each of the target quality assessment methods also includes: a relative assessment method.

[0071] In an optional embodiment, the target quality assessment method includes: a direct assessment method, a relative assessment method and a defect assessment method; the target quality assessment method, the sample image and the description content are input into the quality detection model to obtain the quality assessment result of the target model, including: obtaining at least one sub-content in the description content; the sub-content is the content of the sample image processed and output by the target model of the corresponding version; for each sub-content, the target quality assessment method, the sample image and the sub-content are input into the quality detection model to obtain the quality assessment result of the target model of the corresponding version of the sub-content.

[0072] The relative evaluation method is used to compare the image description quality of different models. Different models can refer to models of different versions or models of different structures. When the target quality evaluation method does not include the relative evaluation method, the description content is the sub-content output by one target model. When the target quality evaluation method includes the relative evaluation method, the description content refers to the sub-content output by multiple target models.

[0073] The sub-content may refer to the content of a version of the target model processing and outputting the sample image. Different sub-contents are output by different versions of the target model. Different sub-contents correspond to different versions of the target model. Each sub-content may be combined with the sample image to obtain a detection object of the image description quality of the target model of the version corresponding to the sub-content.

[0074] Multiple sub-contents may be input into the quality detection model simultaneously, or may be input into the quality detection model separately in batches.

[0075] Exemplarily, the number of sub-contents is n, and the relative evaluation method has n quality evaluation results. In one example, the sample image and each sub-content are added to the prompt template to obtain input data, and the input data is input into the quality detection model to obtain n quality evaluation results, wherein each sub-content can be separated by a specific symbol. In another example, the sample image and the i-th sub-content are added to the prompt template to obtain the i-th input data, and the i-th input data is input into the quality detection model to obtain the i-th quality evaluation result.

[0076] The relative evaluation method is combined with other evaluation methods except the relative evaluation method to obtain corresponding quality evaluation results, where other evaluation methods include direct evaluation methods and / or defect evaluation methods. Among the n quality evaluation results of the relative evaluation method, each quality evaluation result includes j quality evaluation results corresponding to other evaluation methods, where j is the number of other evaluation methods included in the target quality evaluation method. The relative evaluation method cannot obtain evaluation results alone, and needs to be combined with other evaluation methods to obtain evaluation results.

[0077] In one example, for each other evaluation method, the other evaluation method, sample image and each sub-content are added to the prompt template to obtain input data, and the input data is input into the quality detection model to obtain n quality evaluation results of the other evaluation method. In another example, for each other evaluation method, the other evaluation method, sample image and i-th sub-content are added to the prompt template to obtain the i-th input data of the other evaluation method, and the i-th input data of the other evaluation method is input into the quality detection model to obtain the i-th quality evaluation result of the other evaluation method.

[0078] For each sub-content, the quality evaluation results of multiple other evaluation methods can be weighted and summed to obtain the evaluation result corresponding to the sub-content. By comparing the evaluation results corresponding to each sub-content, the sub-content with the best image description quality can be determined, and the version corresponding to the best sub-content is the version with the best image description quality.

[0079] Optionally, for each other evaluation method, k quality detection models can be used for quality detection, and the k quality detection models respectively output the evaluation results of the other evaluation method. The k quality detection models output a total of k evaluation results. These k evaluation results can be subjected to outlier elimination and summed and averaged to obtain the i-th quality evaluation result of the other evaluation method.

[0080] Optionally, for the defect assessment method, each assessment rule outputs an assessment result, and a total of m assessment results can be obtained for m assessment rules. m assessment rules, sample images and sub-contents are input into k quality detection models to obtain k*m assessment results. For the m assessment rules, the k assessment results corresponding to the assessment rule can be subjected to outlier elimination and summed and averaged to obtain the assessment result corresponding to the assessment rule. The assessment results of the m assessment rules are then weighted and summed according to the weights of each assessment rule to obtain the assessment result corresponding to the defect assessment method.

[0081] In addition, the sub-contents may be mapped to target models of different structures, thereby comparing the image description qualities of target models of different structures.

[0082] By using a relative evaluation method to perform quality inspection on the sub-contents output by different versions of the target model, we can obtain quality evaluation results of different versions of the target model, and compare different versions of the target model. This can be used in scenarios where quality comparison is required between different versions of models, providing data support for the screening of different versions of models.

[0083] In an optional embodiment, the defect assessment method includes at least one assessment rule, and the quality assessment result corresponding to the defect assessment method includes the assessment results corresponding to each of the assessment rules.

[0084] In the defect assessment method, more detailed assessment rules that reflect defects in image description quality can be set for each field. The assessment rules are particularly suitable for detecting fine-grained and more professional image description quality standards in the target field.

[0085] By configuring the quality assessment results corresponding to the defect assessment method as the assessment results corresponding to each assessment rule, the corresponding assessment results can be determined for each assessment rule, thereby realizing multi-dimensional detection of image description quality, increasing the flexibility and comprehensiveness of image description quality detection, and at the same time, the image description quality can be detected from a finer granularity, which can improve the accuracy of the image description quality, and can also increase the interpretability of the image description quality assessment results, thereby improving the reliability of the image description quality assessment results.

[0086] In an optional embodiment, the sample image is an image of a target domain, the description content belongs to the content of the target domain, the evaluation rules correspond to the target domain, and the quality detection model is a large language model trained with knowledge of the target domain.

[0087] The target field can refer to a specific professional and technical field, such as space design, sports, education, and medical fields.

[0088] In one example, the sample image may be an interior design drawing in the field of space design, the description content may be the design structure and ideas in the interior design drawing, and the evaluation rule may be whether the description content includes the design dimensions in the interior design drawing.

[0089] In one example, the sample image may be a symptom detection image in the medical field, the description content may be the symptom content in the symptom detection image, and the evaluation rule may be whether there is a professional term for the symptom in the description content.

[0090] In one example, the sample image may be a principle explanation diagram in the field of education, the description content may be the text in the principle explanation diagram, and the evaluation rule may be whether the description content contains all the text content in the principle explanation diagram.

[0091] In one example, the sample image may be a disassembled diagram of a motion posture in the sports field, the description content may be the motion posture content in the motion posture screenshot, and the evaluation rule may be whether the description content contains the states of multiple limbs of the motion posture.

[0092] The quality detection model has been supervisedly trained with knowledge of the target domain and has the knowledge of the target domain. It can accurately extract, analyze and process the information of the target domain, thereby improving the accuracy of image description quality detection in the target domain.

[0093] By configuring sample images, description content, target rules and quality detection models as scenarios applied to the target field, the accuracy of image description quality detection in the target field can be improved, and professional and precise image description quality detection can be performed for the target field.

[0094] Figure 3 It is a flowchart of another image description data processing method disclosed in accordance with an embodiment of the present disclosure, which is further optimized and expanded based on the above technical solution, and can be combined with the above optional implementation modes. The image description data processing method is optimized and increased: a step of determining the weight of the target quality assessment method according to the sample image and the description content, and at the same time, inputting each of the target quality assessment methods, the sample image and the description content into the quality detection model to obtain the quality assessment result of the target model, which is specifically as follows: inputting each of the target quality assessment methods, the sample image and the description content into the quality detection model to obtain the assessment result corresponding to each of the target quality assessment methods; determining the quality assessment result of the target model for the sample image according to the weight corresponding to each of the target quality assessment methods and the assessment result corresponding to each of the target quality assessment methods.

[0095] S301: Input a sample image into a target model to be detected for processing, and obtain a description content output by the target model.

[0096] S302: Determine at least one target quality assessment method according to the sample image and the description content.

[0097] S303: Determine a weight of a target quality assessment method according to the sample image and the description content.

[0098] The weight of the target quality assessment method is used to determine the weight of the assessment result of the target quality assessment method, so as to describe the importance of the target quality assessment method in the fusion of the assessment results of the target quality assessment method. Generally, the higher the weight, the higher the accuracy of the assessment result of the corresponding target quality assessment method, that is, the assessment result of the corresponding dimension; the lower the weight, the lower the accuracy of the assessment result of the corresponding target quality assessment method, that is, the assessment result of the corresponding dimension. S302 and S302 can be executed at the same time. The weight of the target quality assessment method is used to characterize the degree of match between the target quality assessment method and the image description quality detection scenario in the target field. In addition, multiple assessment rules are also configured for the aforementioned defect assessment method, and the weight of the target quality assessment method also includes the weight of the assessment rule, and the weight of each assessment rule can also be determined according to the sample image and the description content.

[0099] In one example, the evaluation method detection model can be used to process the sample image and the description content to obtain at least one target quality evaluation method and the weight of each target quality evaluation method. For example, the output of the evaluation method detection model is: direct evaluation method 60% + defect evaluation method 40%. Among them, the target quality evaluation method includes a direct evaluation method and a defect evaluation method. Among them, the weight of the direct evaluation method is 60%, and the weight of the defect evaluation method is 40%.

[0100] There are multiple evaluation rules configured in the defect evaluation method. The evaluation method detection model outputs the target quality evaluation method including the defect evaluation method, and also outputs the evaluation rules used in the defect evaluation method. As in the previous example, the output of the evaluation method detection model is: direct evaluation method 60% + defect evaluation method 40%. The evaluation rules of the defect evaluation method include: dermatology image description evaluation criteria, and the weights of each evaluation rule are the same. The weights of the evaluation rules of the defect evaluation method are the same, for example, it can be 100% / k, where k is the number of evaluation rules.

[0101] In addition, the combination of target quality assessment methods corresponding to multiple fields, the weight of each target quality assessment method in each combination, and the weight of the assessment rule can be preset. In addition to being obtained by the assessment method detection model, the weight of the target quality assessment method in each combination can also be customized by the user, or can be determined based on experimental data.

[0102] S304: Input each of the target quality assessment methods, the sample image and the description content into a quality detection model to obtain an assessment result corresponding to each of the target quality assessment methods.

[0103] S305 : Determine the quality evaluation result of the target model for the sample image according to the weights corresponding to the target quality evaluation methods and the evaluation results corresponding to the target quality evaluation methods.

[0104] The evaluation results corresponding to each target quality evaluation method may be weighted and summed according to the weight corresponding to each target quality evaluation method to obtain the quality evaluation result of the target model for the sample image.

[0105] According to the technical solution disclosed in the present invention, by configuring the importance of evaluation results in different dimensions through weights corresponding to different target quality evaluation methods, and setting the evaluation matching degree of different dimensions for different fields, the evaluation results of image description quality can be accurately and flexibly determined.

[0106] In a specific example, Figure 4 As shown, the process of the image description data processing method may include:

[0107] S401: Input a sample image.

[0108] S402: Generate description content.

[0109] The sample image is input to the target model, and the target model generates description content. When there are multiple target models, each target model has a sub-content, and multiple sub-contents are finally generated as description content.

[0110] S403. Direct assessment.

[0111] S404. Relative evaluation.

[0112] S405. Defect assessment.

[0113] At least one target quality assessment method is selected from the direct assessment method, the relative assessment method and the defect assessment method, and the target quality assessment method, the sample image and the description content are input into at least one quality detection model to obtain multiple assessment results.

[0114] S406, post-processing: weighted averaging, outlier removal and consistency check, etc.

[0115] Post-process multiple evaluation results, including but not limited to weighted averaging, outlier removal, and consistency testing.

[0116] S407: Generate quality assessment results.

[0117] After post-processing, the quality evaluation results of each target model are obtained, and subsequent processing can be performed based on the quality evaluation results of each target model.

[0118] S408, high-quality data reserves.

[0119] S409, model to be optimized.

[0120] S410. Evaluation conclusion.

[0121] Subsequent processing may include but is not limited to high-quality data reserves, evaluation conclusions, and determination of models to be optimized. High-quality data reserves may be data currently used for training and stored as high-quality data to train relevant models. Models to be optimized may refer to models that have been screened out for poor image description content, and such models still require training and iterative optimization. An optimization model iteration strategy may be used to train and optimize the model to be optimized. After iterative optimization of the model to be optimized, the above process may be re-executed to detect whether the model to be optimized still needs to be iteratively optimized. The evaluation conclusions may be directly output to the user, who may then proceed to the next step, such as publishing it online.

[0122] Disadvantages of existing technology:

[0123] High subjectivity: The evaluation results may vary significantly due to the different professional knowledge of the evaluators and the interference of subjective factors, resulting in low consistency and reliability of the evaluation.

[0124] Time-consuming and inefficient: Manual evaluation requires a lot of time and human resources, especially when dealing with large-scale data, the efficiency problem is particularly obvious.

[0125] Lack of domain specificity: General evaluation schemes are often based on widely applicable standards and fail to fully consider the strict requirements of certain fields (such as medicine). For example, in the process of evaluating the quality of image-generated text, general evaluation schemes cannot score the problem of image hallucination and may ignore important quality details.

[0126] The image description data processing method provided by the disclosed embodiment provides a fully automated evaluation method specifically for multimodal large models, which is particularly suitable for medical and health scenarios with extremely high requirements for generation results. The disclosed embodiment integrates multiple evaluation methods and combines strict field-specific standards to comprehensively and objectively evaluate the generation quality and actual application performance, thereby effectively supporting model optimization and strategy iteration.

[0127] Fully automated evaluation process: The automated process replaces traditional manual labeling, significantly improving evaluation efficiency and reducing subjective errors.

[0128] Multiple evaluation methods: The disclosed embodiments provide multiple evaluation methods, including direct evaluation method, relative evaluation method and defect evaluation method, and objectively integrate these evaluation results through post-processing methods to generate reliable key business indicators.

[0129] Key business indicator orientation: Provide key business indicators such as the quality rate and availability rate of generated results, directly guide instructions, strategies, model optimization and practical applications, and enhance practicality and effectiveness.

[0130] Customized evaluation and optimization: Customized evaluation cabinets are developed based on the needs of different health application scenarios. R&D solutions are continuously optimized by establishing a high-quality data reflux reserve and a collection of models to be optimized (bad cases).

[0131] Automated evaluation process

[0132] (1) Business accumulation: The disclosed embodiments integrate a large medical multimodal model, evaluation standards for a variety of medical and health scenarios, and customized solutions. In a multimodal serious evaluation scenario, a combination strategy of different evaluation standards and solutions can be given based on the input of the evaluation background.

[0133] (2) Diversity of evaluation methods: The present disclosure provides three evaluation methods to ensure a comprehensive evaluation of the model generation results:

[0134] Direct evaluation method: Use a quality detection model to evaluate each generated result and quickly measure whether the result achieves the expected effect.

[0135] Relative evaluation method: By comparing the generation results of different models or different versions under the same input, their relative advantages and disadvantages are evaluated to help identify performance differences between models or strategies.

[0136] Defect assessment method: deduct points for potential defects or errors in the generated results to more carefully reflect the shortcomings in the generated quality, which is particularly suitable for capturing requirements in health scenarios.

[0137] (2) Integration of post-processing methods: In order to objectively and impartially integrate the results of the three evaluation methods, the present disclosure provides the following post-processing methods:

[0138] Weighted average: Different weights are assigned to each evaluation method according to its importance and applicability, and the evaluation results are weighted averaged.

[0139] Outlier removal: Call different models for evaluation, remove the highest and lowest scores, and ensure the stability and reliability of the comprehensive score.

[0140] Consistency check: Before comprehensive scoring, check the consistency of the three evaluation results. If the consistency is poor, you can perform multiple rounds of evaluation and then take the average to reduce jitter.

[0141] (3) Generation of key business indicators: After the comprehensive score is generated, the embodiment of the present disclosure calculates key indicators such as the quality rate and availability rate of the generated results according to specific business requirements. These indicators directly reflect the performance of the model in practical applications and provide a reliable basis for model optimization and business decision-making.

[0142] Customized evaluation rules

[0143] Strict rules in medical scenarios: The disclosed embodiments have formulated multiple sets of strict domain-specific evaluation rules specifically for medical and health scenarios. These standards not only cover general quality indicators such as semantic fluency, coherence, and readability, but also pay special attention to the requirement to avoid generating hallucinatory descriptions in graphic applications in health scenarios. Through these strict standards, the disclosed embodiments ensure the reliability and security of the generated results in sensitive fields such as medical care.

[0144] Data reflux and optimization mechanism

[0145] (1) High-quality data return reserve: During the evaluation process, the embodiment of the present disclosure automatically selects high-quality generated results based on the evaluation results and returns them to the data reserve. These high-quality data can be used to further train and optimize the model and improve the overall performance of the model.

[0146] (2) Establishment of Bad Case Set: For the generated results that perform poorly during the evaluation process, the embodiment of the present disclosure classifies them into the bad case set to be optimized. By analyzing these bad cases, the weaknesses and potential improvement points of the model are identified, so as to optimize the model in a targeted manner.

[0147] 4. Overall benefits and technical advantages

[0148] (1) Full automation, reducing manual intervention: The evaluation process of the disclosed embodiment is fully automated, eliminating the reliance on manual annotation and significantly improving evaluation efficiency. The automated process not only saves a lot of human resources, but also effectively reduces evaluation errors caused by subjective factors.

[0149] (2) Multi-level evaluation to ensure the reliability of results: By integrating multiple evaluation methods and strict post-processing methods, the disclosed embodiments can comprehensively and objectively evaluate the generation results of MLLMs. Multi-level evaluation ensures the reliability and fairness of the results, which is particularly suitable for applications in health scenarios that have extremely high requirements for the quality of results.

[0150] (3) Customized optimization to improve model practicality: The customized evaluation criteria and data reflow mechanism provided by the embodiments of the present disclosure make model optimization more targeted and accurate. Through continuous iterative optimization, the actual application effect and business value of the model are significantly improved.

[0151] In a specific example, in the field of Internet medicine, patients often upload medical images (such as pictures of skin lesions) and describe symptoms through online platforms to obtain professional medical interpretations and suggestions. However, the evaluation of image descriptions relies on manual annotation of data samples. The automated evaluation system of the disclosed embodiment can be effectively applied to this scenario to evaluate the effectiveness of current medical image interpretation.

[0152] To evaluate whether the v2 version of the target model is improved over the v1 version of the target model, the data set consists of 100 randomly sampled sample images, 100 sub-contents obtained by inputting the 100 sample images into the v2 target model, and 100 sub-contents obtained by inputting the 100 sample images into the v1 target model.

[0153] Regarding the evaluation rules for defect assessment methods, medical scenarios have more stringent requirements for the accuracy and completeness of graphic descriptions. Take the dermatology evaluation dimension as an example:

[0154] Evaluation rule s1 accuracy evaluation: describe the medical information contained in the sample image presented in the content, such as anatomical structure, pathological characteristics and instrument readings. Among them, s1 = 3 points means the proportion is [90, 100]; s1 = 2 points means the proportion is [80, 90); s1 = 1 means the proportion is [60, 80); s1 = 0 means the proportion is [40, 60); s1 = -1 means the proportion is [20, 40); s1 = -2 means the proportion is [0, 20).

[0155] Evaluation rule s2 Completeness evaluation: The sample image in the description content is fully retained, including all important medical information and details. For example, not only the existence of symptoms is mentioned, but also their size and morphology are described. Among them, s2 = 3 points means the proportion is [90, 100]; s2 = 2 points means the proportion is [80, 90); s2 = 1 means the proportion is [60, 80); s2 = 0 means the proportion is [40, 60); s2 = -1 means the proportion is [20, 40); s2 = -2 means the proportion is [0, 20).

[0156] Evaluation rules s3 Text OCR (Optical Character Recognition) content evaluation: For the text included in the sample image, complete the key points of the paraphrase, etc. For example, test report sheets, medicine box instructions, and medical popular science content. Among them, s3 = 3 points means the proportion is [90, 100]; s3 = 2 points means the proportion is [80, 90); s3 = 1 means the proportion is [60, 80); s3 = 0 means the proportion is [40, 60); s3 = -1 means the proportion is [20, 40); s3 = -2 means the proportion is [0, 20).

[0157] Evaluation rule s4 Redundancy evaluation: No redundant information, condense the content of the sample image. For example, accurately describe the symptoms shown in the picture, without inferring sensory descriptions such as pain or itching. Among them, s4 = 3 points means the proportion is [90, 100]; s4 = 2 points means the proportion is [80, 90); s4 = 1 means the proportion is [60, 80); s4 = 0 means the proportion is [40, 60); s4 = -1 means the proportion is [20, 40); s4 = -2 means the proportion is [0, 20).

[0158] Evaluation rule s5 Medical language professionalism evaluation: whether the given descriptive words are consistent with the professional terms of the target field of the current period. Among them, s4 = 3 points means the proportion is [90, 100]; s4 = 2 points means the proportion is [80, 90); s4 = 1 means the proportion is [60, 80); s4 = 0 means the proportion is [40, 60); s4 = -1 means the proportion is [20, 40); s4 = -2 means the proportion is [0, 20).

[0159] Among them, the above-mentioned scores can be scores manually given by professional users in the target field, or, a substantial analysis can be conducted on the description content, and the text that meets the requirements can be extracted, and the proportion of the text volume to the total text volume can be calculated. The corresponding score can be determined according to the interval to which the proportion belongs; keywords can be extracted, and the proportion of the number of keywords to the number of entity words included in the description content can be counted, and the corresponding score can be determined according to the interval to which the proportion belongs.

[0160] Take a single image as an example:

[0161] Input image: A picture of "common warts on the scalp symptoms pictures"

[0162] Output description:

[0163] v1 Image Description: This image shows an area on the scalp with obvious symptoms of dandruff and folliculitis. Dandruff is the shedding of keratin cells on the surface of the scalp, which is usually white or gray, while folliculitis is an inflammatory reaction around the hair follicles that may appear as redness, swelling, pain, or itching.

[0164] Analysis: The content described in the red text is very different from the content of the picture. There is a subjective diagnosis of [folliculitis] and a sensory description illusion of [redness, swelling, pain or itching]. There is more wrong information than correct information.

[0165] v2 Image Description: This image shows symptoms of common warts on the scalp. The rash in the image appears round or oval, with a rough, darker color and surrounded by fine hairs.

[0166] Analysis: The symptoms of the scalp area in the picture are described truthfully and accurately. Although a subjective diagnosis of [common warts] appears, it accounts for a small proportion of the description and has little impact on the overall picture.

[0167] The same image is input into the v1 version of the large language model and the v2 version of the same large language model.

[0168] The medical big model (i.e., quality detection model) obtained through supervised training of medical knowledge selects the target quality assessment method as the most appropriate assessment scheme from the built-in alternative quality assessment methods of health scenarios and multiple assessment rules of defect assessment methods according to the assessment objectives.

[0169] For example, the medical big model provides an evaluation plan as follows:

[0170] [Direct assessment + defect assessment] are assigned weights of 60% and 40% respectively. The evaluation rules for defect assessment are [Dermatology Image Description Evaluation Standard], and each evaluation rule has an equal weight.

[0171] The sample image, description content v1 and description content v2 are input into the quality inspection model to obtain the evaluation results corresponding to the defect evaluation of v1, the evaluation results corresponding to the direct evaluation of v1, the evaluation results corresponding to the defect evaluation of v2 and the evaluation results corresponding to the direct evaluation of v2.

[0172] Take the single-argument evaluation of description content v1 as an example:

[0173] The evaluation result corresponding to the defect evaluation of v1 is: the score si of the evaluation standard i: for example, accuracy evaluation s1 = -1; completeness evaluation s2 = -2; text OCR content evaluation s3 = 3; redundancy evaluation s4 = 0; medical language professional evaluation s5 = 3. n is the number of evaluation standards. The value of i is [1, n]. The evaluation result corresponding to the defect evaluation is:

[0174]

[0175] The direct evaluation result of v1 is En = -1

[0176] The quality evaluation result of v1 score = 60% * (-1) + 40% * 0.6 = -0.36

[0177] N rounds of evaluation are performed based on a total of N sample images in the evaluation dataset. In each round, the medical multimodal large prediction model selects the scoring criteria based on the input image discrimination. In order to reduce the jitter of the evaluation, multiple MLLMs can be selected in each round to take the average.

[0178] The quality assessment results of v1 for N = 100 sample images are:

[0179]

[0180] The value of j is [1, N]. After 100 rounds of evaluation on the two versions of the evaluation set, the final score is obtained:

[0181] Score_v1=0.35, Score_v1=1.72

[0182] Evaluation conclusion:

[0183] The description effect of version v2 is better than that of version v1, so the iteration of the v2 strategy is effective and can be used for full mining iterative graphic descriptions and applied online.

[0184] The disclosed embodiments significantly improve the efficiency of output evaluation of multimodal large model applications in medical scenarios, improve strategy accuracy, and reduce manual intervention, thereby supporting product development and iteration speed, and can provide more reliable and accurate high-quality descriptions, significantly improving the security of Internet medical care and the effectiveness of decision support. The disclosed embodiments focus on business-oriented evaluation solutions, especially for the high standards in the medical and health field, and provide strict customized standards and multi-level evaluation methods to better meet the needs of specific business scenarios. In addition, the disclosed embodiments reduce manual intervention, shorten evaluation time, and significantly improve R&D efficiency.

[0185] According to an embodiment of the present disclosure, Figure 5 1 is a structural diagram of an image description data processing device in an embodiment of the present disclosure, and the present disclosure embodiment is applicable to the case where the quality of the model output result of the image description is determined. The device is implemented by software and / or hardware, and is specifically configured in an electronic device with certain data computing capabilities.

[0186] like Figure 5 The image description data processing device 500 shown in the figure comprises: an image description output module 501, an evaluation method determination module 502 and a quality evaluation determination module 503.

[0187] The image description output module 501 is used to input the sample image into the target model to be detected for processing, and obtain the description content output by the target model;

[0188] An evaluation method determination module 502, configured to determine at least one target quality evaluation method according to the sample image and the description content;

[0189] The quality assessment determination module 503 is used to determine the quality assessment result of the target model for the sample image according to the sample image and the description content by adopting each of the target quality assessment methods.

[0190] The disclosed embodiment determines at least one target quality assessment method based on sample images and description content, and can determine a quality assessment method adapted to the corresponding image content to perform quality assessment on the target model. It can adapt to the assessment targets of different image contents, and specifically assess the description quality of the large model in specific content. It can comprehensively and accurately assess the image description quality of the large model, which is beneficial to selecting high-quality models and controlling the quality of the online model in model iteration, thereby improving image processing quality.

[0191] Optionally, the quality assessment determination module 503 includes: a quality detection input unit, used to input each of the target quality assessment methods, the sample image and the description content into a quality detection model to obtain a quality assessment result of the target model.

[0192] Optionally, the image description data processing device further includes:

[0193] A method weight determination module is used to determine the weight of the target quality assessment method according to the sample image and the description content, including:

[0194] The quality detection input unit comprises:

[0195] A single-mode evaluation subunit, used for inputting each of the target quality evaluation modes, the sample image and the description content into a quality detection model to obtain an evaluation result corresponding to each of the target quality evaluation modes;

[0196] The evaluation result fusion subunit is used to determine the quality evaluation result of the target model for the sample image according to the weights corresponding to each of the target quality evaluation methods and the evaluation results corresponding to each of the target quality evaluation methods.

[0197] Optionally, the quality detection input unit includes:

[0198] A multi-model evaluation subunit, configured to use at least one quality detection model to obtain, according to each of the target quality evaluation methods, the sample image and the description content, an intermediate result corresponding to each of the target quality evaluation methods output by each of the quality detection models;

[0199] An intermediate result fusion subunit, used to determine the evaluation results corresponding to each of the target quality evaluation methods according to the intermediate results corresponding to each of the target quality evaluation methods output by each of the quality detection models;

[0200] The evaluation result determination subunit is used to determine the quality evaluation result of the target model for the sample image according to the evaluation results corresponding to each of the target quality evaluation methods.

[0201] Optional, intermediate result fusion subunit, specifically used for:

[0202] Eliminate abnormal values ​​from the intermediate results corresponding to each of the target quality assessment methods output by each of the quality detection models;

[0203] For each target quality assessment method, the intermediate results corresponding to the remaining target quality assessment methods are averaged to obtain an assessment result corresponding to the target quality assessment result.

[0204] Optionally, each of the target quality assessment methods includes at least one of the following: a direct assessment method, a relative assessment method, and a defect assessment method.

[0205] Optionally, the target quality assessment method includes: a direct assessment method, a relative assessment method and a defect assessment method; other assessment methods include: a direct assessment method and a defect assessment method;

[0206] The quality detection input unit comprises:

[0207] A sub-content acquisition sub-unit, used to acquire at least one sub-content in the description content; the sub-content is the content output by the target model of the corresponding version processing the sample image;

[0208] The sub-content quality detection sub-unit is used to input each of the target quality assessment methods, the sample image and the sub-content into a quality detection model for each of the sub-contents to obtain a quality assessment result of the target model of the corresponding version of the sub-content.

[0209] Optionally, the defect assessment method includes at least one assessment rule, and the quality assessment result corresponding to the defect assessment method includes the assessment results corresponding to each of the assessment rules.

[0210] Optionally, the sample image is an image of a target domain, the description content belongs to the content of the target domain, the evaluation rules correspond to the target domain, and the quality detection model is a large language model trained with knowledge of the target domain.

[0211] Optionally, the image description data processing device further includes:

[0212] A multi-image quality detection module, configured to obtain, after determining the quality evaluation result of the target model for the sample image, quality evaluation results corresponding to the target model for a plurality of other images;

[0213] The multi-image evaluation result fusion module is used to determine the quality evaluation information of the target model according to the quality evaluation result of the sample image and the quality evaluation results corresponding to each of the other images.

[0214] Optionally, the evaluation method determination module includes:

[0215] An evaluation method determination unit is used to input the sample image and the description content into an evaluation method detection model for processing, and output at least one target quality evaluation method. The evaluation method detection model is trained with knowledge of the target field, and the sample image and the description content correspond to the target field.

[0216] The above-mentioned image description data processing device can execute the image description data processing method provided by any embodiment of the present disclosure, and has corresponding functional modules and beneficial effects for executing the image description data processing method.

[0217] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0218] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0219] Figure 6 A schematic area diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0220] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the device 600 instructions can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0221] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0222] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the image description data processing method. For example, in some embodiments, the image description data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the image description data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the image description data processing method in any other appropriate manner (e.g., by means of firmware).

[0223] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard objects (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0224] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / instructions specified in the flow chart and / or the area diagram. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0225] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0226] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0227] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0228] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0229] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0230] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, instruction systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as model training.

[0231] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution provided by this disclosure can be achieved, and this document does not limit this.

[0232] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for processing image description data, comprising: Inputting the sample image into the target model to be detected for processing, and obtaining the description content output by the target model; Determining at least one target quality assessment method according to the sample image and the description content; Each of the target quality assessment methods is adopted to determine the quality assessment result of the target model for the sample image according to the sample image and the description content.

2. The method according to claim 1, wherein: The adopting each of the target quality assessment methods to determine the quality assessment result of the target model for the sample image according to the sample image and the description content includes: Each of the target quality assessment methods, the sample image and the description content is input into a quality detection model to obtain a quality assessment result of the target model.

3. The method according to claim 2, further comprising: Determining a weight of a target quality assessment method according to the sample image and the description content; The step of inputting each of the target quality assessment methods, the sample image and the description content into a quality detection model to obtain a quality assessment result of the target model includes: Inputting each of the target quality assessment methods, the sample image and the description content into a quality detection model to obtain an assessment result corresponding to each of the target quality assessment methods; The quality assessment result of the target model for the sample image is determined according to the weights corresponding to the target quality assessment methods and the assessment results corresponding to the target quality assessment methods.

4. The method according to claim 2, wherein: The step of inputting each of the target quality assessment methods, the sample image and the description content into a quality detection model to obtain a quality assessment result of the target model includes: Using at least one quality detection model, according to each of the target quality assessment methods, the sample image and the description content, obtain an intermediate result corresponding to each of the target quality assessment methods output by each of the quality detection models; Determine the evaluation results corresponding to each of the target quality evaluation methods according to the intermediate results corresponding to each of the target quality evaluation methods output by each of the quality detection models; According to the evaluation results corresponding to each of the target quality evaluation methods, the quality evaluation result of the target model for the sample image is determined.

5. The method according to claim 4, wherein: Determining the evaluation results corresponding to the target quality evaluation methods according to the intermediate results corresponding to the target quality evaluation methods output by the quality detection models includes: Eliminate abnormal values ​​from the intermediate results corresponding to each of the target quality assessment methods output by each of the quality detection models; For each target quality assessment method, the intermediate results corresponding to the remaining target quality assessment methods are averaged to obtain an assessment result corresponding to the target quality assessment result.

6. The method according to claim 2, wherein: Each of the target quality assessment methods includes: a direct assessment method and / or a defect assessment method, and each of the target quality assessment methods also includes: a relative assessment method.

7. The method according to claim 6, wherein: The target quality assessment methods include: direct assessment method, relative assessment method and defect assessment method; The step of inputting each of the target quality assessment methods, the sample image and the description content into a quality detection model to obtain a quality assessment result of the target model includes: Acquire at least one sub-content in the description content; the sub-content is the content output by the target model of the corresponding version processing the sample image; For each of the sub-contents, each of the target quality assessment methods, the sample image and the sub-content is input into a quality detection model to obtain a quality assessment result of the target model of the version corresponding to the sub-content.

8. The method according to claim 6, wherein: The defect assessment method includes at least one assessment rule, and the quality assessment result corresponding to the defect assessment method includes the assessment results corresponding to each of the assessment rules.

9. The method according to claim 8, wherein: The sample image is an image of a target domain, the description content belongs to the content of the target domain, the evaluation rule corresponds to the target domain, and the quality detection model is a large language model trained with knowledge of the target domain.

10. The method according to claim 1, after determining the quality evaluation result of the target model for the sample image, further comprising: Obtaining quality assessment results corresponding to the target model for multiple other images; The quality assessment information of the target model is determined according to the quality assessment result of the sample image and the quality assessment results corresponding to each of the other images.

11. The method according to claim 1, wherein: The determining, according to the sample image and the description content, at least one target quality assessment method comprises: The sample image and the description content are input into an evaluation method detection model for processing, and at least one target quality evaluation method is output. The evaluation method detection model is trained with knowledge of the target field, and the sample image and the description content correspond to the target field.

12. An image description data processing device, comprising: An image description output module is used to input a sample image into a target model to be detected for processing, and obtain a description content output by the target model; An evaluation method determination module, used to determine at least one target quality evaluation method according to the sample image and the description content; The quality assessment determination module is used to adopt each of the target quality assessment methods to determine the quality assessment result of the target model for the sample image according to the sample image and the description content.

13. The device according to claim 12, wherein: The quality assessment determination module comprises: The quality detection input unit is used to input each of the target quality assessment methods, the sample image and the description content into the quality detection model to obtain the quality assessment result of the target model.

14. The apparatus according to claim 13, further comprising: A method weight determination module is used to determine the weight of the target quality assessment method according to the sample image and the description content, including: The quality detection input unit comprises: A single-mode evaluation subunit, used for inputting each of the target quality evaluation modes, the sample image and the description content into a quality detection model to obtain an evaluation result corresponding to each of the target quality evaluation modes; The evaluation result fusion subunit is used to determine the quality evaluation result of the target model for the sample image according to the weights corresponding to each of the target quality evaluation methods and the evaluation results corresponding to each of the target quality evaluation methods.

15. The device according to claim 13, wherein: The quality detection input unit comprises: A multi-model evaluation subunit, configured to use at least one quality detection model to obtain, according to each of the target quality evaluation methods, the sample image and the description content, an intermediate result corresponding to each of the target quality evaluation methods output by each of the quality detection models; An intermediate result fusion subunit, used to determine the evaluation results corresponding to each of the target quality evaluation methods according to the intermediate results corresponding to each of the target quality evaluation methods output by each of the quality detection models; The evaluation result determination subunit is used to determine the quality evaluation result of the target model for the sample image according to the evaluation results corresponding to each of the target quality evaluation methods.

16. The device according to claim 15, wherein: The intermediate result fusion subunit is specifically used for: Eliminate abnormal values ​​from the intermediate results corresponding to each of the target quality assessment methods output by each of the quality detection models; For each target quality assessment method, the intermediate results corresponding to the remaining target quality assessment methods are averaged to obtain an assessment result corresponding to the target quality assessment result.

17. The device according to claim 13, wherein: The target quality assessment methods include at least one of the following: a direct assessment method, a relative assessment method, and a defect assessment method.

18. The device according to claim 17, wherein: The target quality assessment methods include: direct assessment method, relative assessment method and defect assessment method; other assessment methods include: direct assessment method and defect assessment method; The quality detection input unit comprises: A sub-content acquisition sub-unit, used to acquire at least one sub-content in the description content; the sub-content is the content output by the target model of the corresponding version processing the sample image; The sub-content quality detection sub-unit is used to input each of the target quality assessment methods, the sample image and the sub-content into a quality detection model for each of the sub-contents to obtain a quality assessment result of the target model of the corresponding version of the sub-content.

19. The device according to claim 17, wherein: The defect assessment method includes at least one assessment rule, and the quality assessment result corresponding to the defect assessment method includes the assessment results corresponding to each of the assessment rules.

20. The device according to claim 19, wherein The sample image is an image of a target domain, the description content belongs to the content of the target domain, the evaluation rule corresponds to the target domain, and the quality detection model is a large language model trained with knowledge of the target domain.

21. The apparatus according to claim 12, further comprising: A multi-image quality detection module, configured to obtain, after determining the quality evaluation result of the target model for the sample image, quality evaluation results corresponding to the target model for a plurality of other images; The multi-image evaluation result fusion module is used to determine the quality evaluation information of the target model according to the quality evaluation result of the sample image and the quality evaluation results corresponding to each of the other images.

22. The device according to claim 12, wherein: The evaluation method determination module includes: An evaluation method determination unit is used to input the sample image and the description content into an evaluation method detection model for processing, and output at least one target quality evaluation method. The evaluation method detection model is trained with knowledge of the target field, and the sample image and the description content correspond to the target field.

23. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the image description data processing method according to any one of claims 1 to 11.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the image description data processing method according to any one of claims 1-11.

25. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the image description data processing method according to any one of claims 1 to 11.

Images