Detection method, system and equipment based on large model and storage medium

Abstract

The invention provides a detection method, system and device based on a large model and a storage medium, and relates to the technical field of computers, in particular to the technical fields of data processing, artificial intelligence and the like. According to the specific implementation scheme, a detection instruction is received, wherein the detection instruction comprises to-be-detected content; determining a detection type based on the detection indication, and calling a multi-modal large model based on the detection type; detecting the to-be-detected content based on calling by using the multi-modal large model; receiving a detection result returned by the multi-modal large model; and performing format arrangement on the detection result, and sending the detection result after format arrangement. According to the invention, multiple types of automatic detection can be provided based on a unified detection platform.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to technical fields such as data processing and artificial intelligence. Background Art

[0002] With the rapid development of artificial intelligence (AI), especially the emergence of large language models and large image generation models, more and more business areas are beginning to integrate these advanced AI capabilities. These large models can usually handle complex tasks, such as image recognition, text-to-image generation, etc.

[0003] However, despite the powerful capabilities of large models, the data they generate does not always fully meet expectations and may have quality issues, such as inconsistency with descriptions, poor visual effects, lack of creativity, etc. Therefore, it is necessary to detect the data generated by large models and / or the data annotated by the annotation team to achieve data screening, evaluation, scoring, etc. Summary of the invention

[0004] The present disclosure provides a large model-based detection method, system, device and storage medium.

[0005] According to one aspect of the present disclosure, a large model-based detection method is provided, comprising:

[0006] Receiving a detection instruction, wherein the detection instruction includes content to be detected;

[0007] Determining a detection type based on the detection indication, and calling a multimodal large model based on the detection type;

[0008] Utilizing the multimodal large model to detect the content to be detected based on the call;

[0009] Receive the detection results returned by the multimodal large model;

[0010] The test results are formatted and sent.

[0011] According to another aspect of the present disclosure, a large model-based detection system is provided, comprising a Web application framework and a multimodal large model, wherein:

[0012] The web application framework is used to receive a detection indication from a client, the detection indication including content to be detected; determine a detection type based on the detection indication, and call the multimodal large model based on the detection type; and format the detection result returned by the multimodal large model, and return the formatted detection result to the client;

[0013] The multimodal large model is used to detect the content to be detected based on the call and return the detection result to the Web application framework.

[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0019] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0020] The disclosed embodiment proposes a detection method based on a large model, which receives a detection indication, determines the detection type based on the detection indication, and calls a multimodal large model based on the detection type to perform a corresponding type of detection; after receiving the detection result of the large model, the detection result can also be formatted and the formatted detection result can be fed back. In this way, automatic content detection based on a multimodal large model can be achieved, and a unified detection platform can be provided to the outside world.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0023] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure;

[0024] Figure 2 is a flow chart of an implementation of a detection method according to an embodiment of the present disclosure;

[0025] Figure 3 is a schematic diagram of an implementation framework of a detection method according to an embodiment of the present disclosure;

[0026] Figure 4 is a warning picture according to an embodiment of the present disclosure;

[0027] Figure 5 is a schematic structural diagram of a large model-based detection system 500 according to an embodiment of the present disclosure;

[0028] Figure 6 A schematic block diagram of an example electronic device 600 that may be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0029] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0030] The “and / or” in the embodiments of the present disclosure indicates that three relationships may exist. For example, A and / or B may indicate the following three situations: A exists alone, A and B exist at the same time, and B exists alone. The term “at least one” herein indicates any combination of at least two of any one or more of a plurality of types. For example, at least one of A, B, and C may indicate any one or more elements selected from the set consisting of A, B, and C. The terms “first” and “second” herein refer to and distinguish between multiple similar technical terms, and do not mean to limit the order or to limit the meaning to only two. For example, the first feature and the second feature refer to two types / two features. The first feature may be one or more, and the second feature may also be one or more.

[0031] With the rapid development of artificial intelligence, especially the emergence of large language models and large image generation models, more and more business areas are beginning to integrate these advanced AI capabilities. These large models can usually handle complex tasks, such as image recognition, text-to-image generation, etc.

[0032] In the image generation scenario, these models can generate corresponding images based on text descriptions, or perform image style conversion, image restoration, etc. The generated images can be used in various applications, such as intelligent content, smart cities, smart transportation, smart industry and other business lines.

[0033] However, despite the powerful capabilities of large models, the data they generate are not always completely consistent with expectations and may have quality issues such as inconsistency with descriptions, poor visual effects, lack of creativity, etc. Therefore, it is necessary to test the data generated by large models to detect the quality of the data generated by large modules.

[0034] In the prior art, manual review is usually used, which greatly increases labor costs, reduces evaluation efficiency, and has low evaluation quality.

[0035] For example, for the quality assessment of AI Wenshengtu images, the images generated by the Wenshengtu large model need to be manually reviewed one by one, and scored according to the degree of match between the prompt and the generated image to evaluate the image quality. Manual review of AI images generated by the Wenshengtu model is not only time-consuming and laborious, but also carries the risk of inconsistent evaluation results due to differences in personal standards and understanding. At the same time, long-term review may cause fatigue and distraction, further affecting the accuracy of the evaluation.

[0036] For example, for the on-site project algorithm scenario error (badcase) mining (referred to as algorithm scenario error mining), the alarms generated by real cameras using online algorithms need to be manually reviewed one by one to determine whether the algorithm scenario definition corresponding to the alarm data can be defined as a badcase. This method has high labor costs, and manual evaluation is prone to fatigue when the amount of data is large, and the definitions of more than 100 algorithm scenarios also require a long time to be familiar with.

[0037] For example, in the scenario where the image annotation team reviews the image annotation, after the data annotator completes the data annotation, the annotation needs to be manually reviewed to verify the accuracy of the annotation. The human review cost is high in this process, and the results of manual verification will be affected by personal subjective factors, resulting in inconsistency in annotation.

[0038] In short, the above methods are all manual observation and review, and the main disadvantages of manual work are low efficiency, high cost, and strong subjectivity. In the relevant technology, there is no unified detection platform for multiple detection types to achieve automated detection of multiple detection types.

[0039] Based on the above situation, the embodiments of the present disclosure propose a detection system and method, which can realize various types of detection based on the capabilities of a multimodal large model, thereby realizing automated detection, reducing human review costs, and improving detection accuracy.

[0040] Figure 1 is a schematic diagram of an application scenario according to an embodiment of the present disclosure, such as Figure 1As shown, the application scenario diagram of the embodiment of the present disclosure may include but is not limited to a client 110 and a detection system 120, and the client 110 and the detection system 120 may communicate with each other through any type of wired or wireless network. Specifically, the client 110 may be used to receive a detection instruction from a user, and send the detection instruction from the user to the detection system 120; the detection system 120 may receive the detection instruction sent by the client, and detect the content to be detected based on the detection instruction, and return the detection results to the client 110 after collating them. Among them, the client 110 proposed in the embodiment of the present disclosure includes but is not limited to electronic devices such as mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, game consoles, e-book readers, multimedia playback devices, wearable devices, etc.; the detection system 120 may include an electronic device or server for providing automated detection services for the client 110. In addition, the embodiment of the present disclosure does not specifically limit the number of behavior clients 110. For example, the application scenario diagram of the embodiment of the present disclosure may include one or more clients 110.

[0041] The disclosed embodiment proposes a detection method based on a large model. Figure 2 is a flow chart of an implementation of a large model-based detection method according to an embodiment of the present disclosure, including:

[0042] S210, receiving a detection instruction, where the detection instruction includes content to be detected;

[0043] S220, determining a detection type based on the detection indication, and calling a multimodal large model based on the detection type;

[0044] S230, using the multimodal large model to detect the content to be detected based on the call;

[0045] S240, receiving the detection result returned by the multimodal large model;

[0046] S250: format the test result and send the formatted test result.

[0047] The disclosed embodiment can receive a detection instruction sent by a user from a client, determine the detection type based on the detection instruction, and call the multimodal large model to perform the corresponding type of detection based on the detection type; after receiving the detection result of the large model, the detection result can also be formatted and the formatted detection result can be fed back to the client. In this way, automatic content detection based on the multimodal large model can be realized, and a unified detection platform can be provided to the outside world.

[0048] In some implementations, the embodiments of the present disclosure may be applied to a detection system including a Web application framework and a multimodal large model. When the detection result is used to implement the detection method of the present solution, the detection method may specifically include:

[0049] The web application framework receives a detection instruction from the client, where the detection instruction includes content to be detected;

[0050] The web application framework determines a detection type based on the detection indication, and calls a multimodal large model based on the detection type;

[0051] The multimodal large model detects the content to be detected based on the call and returns the detection result to the Web application framework;

[0052] The Web application framework formats the detection results and returns the formatted detection results to the client.

[0053] The above example constructs a detection system that includes a Web application framework and a multimodal large model, and this detection system can be used to perform various types of detection.

[0054] In some embodiments, the web application framework includes FastAPI, which is a high-performance web framework that provides the ability to quickly build APIs in a simple way. FastAPI supports asynchronous programming, making it easier to handle a large number of concurrent requests.

[0055] In some embodiments, the detection type includes at least one of AI Wensheng graph data quality assessment, algorithm scenario error mining, and image annotation assessment. Therefore, the detection method proposed in the embodiment of the present disclosure can provide multiple types of automated detection based on a unified detection platform, such as quality assessment of images generated by AI Wensheng graph models, mining of error cases in alarm data detected by other large models in the algorithm scenario of the field project, and evaluation of data annotated by the image annotation team.

[0056] In some implementations, embodiments of the present disclosure pre-train and deploy a multi-modal large model that can provide detection capabilities for at least two of the above detection types;

[0057] At least two application interfaces (such as API interfaces) are established, each application interface corresponding to a detection capability of a detection type.

[0058] For example, the embodiment of the present disclosure pre-trains a multimodal large model, which can provide AI text image data quality assessment, algorithm scenario error mining, and image annotation assessment; and establishes three API interfaces, each of which corresponds to the above three detection capabilities. In addition, the embodiment of the present disclosure deploys the multimodal large model, such as deploying the multimodal large model in a graphics card device.

[0059] The API interface can support the Web application framework to call the multimodal large model. In one example, the Web application framework calls the application interface based on the detection type to realize the call of the multimodal large model; wherein the application interface called by the Web application framework corresponds to the detection capability of the detection type. By establishing corresponding application interfaces for different detection capabilities of the multimodal large model, it is convenient for the Web application framework to call different capabilities of the multimodal large model, thereby simplifying the model calling process.

[0060] In some examples, open source multimodal large models are used, and the open source multimodal large models are trained and deployed offline, such as using data from business-related algorithm scenarios for data cleaning, labeling, self-training, and model parameter adjustment. Through these operations, the recognition accuracy of the model in specific scenarios is improved.

[0061] In addition, based on the AI-generated images provided by the intelligent content business line, badcase images of algorithm scenarios, and image data from the data annotation team, targeted prompt adjustments can be made to the multi-model large model to accurately optimize the multi-model large model's ability to understand different needs, thereby achieving better recognition effects.

[0062] In some implementations, the process may further include before step S220:

[0063] Extract a table in a predetermined format;

[0064] Extracting content of a specified column from the table in the predetermined format according to a predetermined strategy, wherein the content of the specified column includes an image and / or text;

[0065] The extracted image and / or text content is stored in a local service path, and the extracted image and / or text content is converted into a format.

[0066] The above process extracts images and / or texts related to the detection content, providing conditions for the subsequent calling of a multimodal large model for detection.

[0067] In some embodiments, using the FastAPI asynchronous framework, the disclosed embodiments can integrate multiple detection capabilities of a multimodal large model into the system. When a user sends the content to be detected through the client, data upload can be supported in a table in a predetermined format. For example, the detection instruction sent by the user through the client contains a table in a predetermined format, and the table in the predetermined format contains multiple contents to be detected. Taking the AI ​​text image data quality assessment as an example, the user (ie, the business party) can organize the pictures to be reviewed, the prompt used by the text image model when generating the picture, and other information into a table in a predetermined format (such as an Excel table). By setting a table in a predetermined format, it is convenient for users to upload the content to be detected, and it is convenient for the Web application framework to save and extract operations such as the content to be detected.

[0068] Accordingly, the Web application framework extracts the content to be detected, including:

[0069] The web application framework extracts the table in the predetermined format;

[0070] The web application framework extracts content of a specified column from a table of a predetermined format according to a predetermined strategy, where the content of the specified column includes an image and / or text;

[0071] The web application framework stores the extracted image and / or text content in a local service path and performs format conversion on the extracted image and / or text content, thereby preparing for calling the multimodal large model.

[0072] By setting a table of a predetermined format and a predetermined strategy, it is possible to facilitate the Web application framework to extract content for detection from the table, thereby improving the speed and accuracy of extracting information.

[0073] In some implementations, corresponding prompt word templates may be generated in advance according to different detection types, and the content of the prompt word template is related to the detection content, scoring criteria, output result format, and other information corresponding to the detection type. The disclosed embodiment generates and saves prompt word templates for different detection types in advance, and when calling the multimodal large model, a corresponding prompt may be generated according to the prompt word template corresponding to the current detection type and the extracted content, and the generated prompt is used to call the multimodal large model to perform the detection task.

[0074] Taking the use of a Web application framework to call a multimodal big model as an example, when calling the multimodal big model, the Web application framework can generate a corresponding prompt based on the detection type and the content to be detected, and input the prompt into the multimodal big model, and the multimodal big model performs detection and evaluation based on the prompt.

[0075] In some examples, the FastAPI service uses the openpyxl library and the pandas library to extract all the contents in the Excel table, and extracts the image and text content of the specified column according to the strategy. All images will be stored in the local service path, and the extracted images will be converted to base64 format to prepare for calling the open source multimodal large model.

[0076] In some implementations, calling a multimodal large model based on the detection type includes:

[0077] The Web application framework obtains a pre-saved prompt word template (prompt template) corresponding to the detection type according to the detection type;

[0078] The web application framework generates prompt words for instructing the multi-model large model to perform detection according to the prompt word template and the extracted content;

[0079] The web application framework calls the multimodal big model and sends the generated prompt words to the multimodal big model.

[0080] By pre-setting prompt templates corresponding to various detection types, and generating a prompt matching the detection type according to the prompt template when the Web application framework calls the multimodal large model, the prompt required for the multimodal large model to perform detection can be quickly generated, and the generated prompt is unified and standardized.

[0081] For example, the Web application framework can generate multiple prompt words for instructing the multi-model large model to perform detection based on the prompt word template and the extracted multiple contents to be detected, and each prompt word is used to instruct the multi-model large model to detect a content to be detected.

[0082] The multimodal large model detects each piece of content to be detected according to the received prompt words; for example, the Web application framework generates multiple prompt words for multiple pieces of content to be detected, and sends the multiple prompt words to the multimodal large model;

[0083] After detection, the multimodal large model returns the detection results of each content to be detected to the Web application framework according to at least one of the output formats and output examples required by each prompt word. The returned detection results can be in JSON format, and the detection results in this JSON format meet the output format and / or output examples required in the prompt.

[0084] After receiving the detection results of multiple contents to be detected returned by the multimodal large model, the Web application framework formats the detection results and returns the formatted detection results to the client, for example, including:

[0085] The web application framework performs text segmentation on the detection results of each piece of content to be detected to obtain at least one of a detection score and a scoring reason;

[0086] The web application framework writes at least one of the detection score and the scoring reason for each content to be detected into a table of a predetermined format to obtain a table filled with the detection results;

[0087] The web application framework returns the form filled with the detection results to the client.

[0088] The detection results returned by the large model are converted into a unified format through the web application framework, and the corresponding information is written into a table, which can facilitate data transmission and user viewing.

[0089] Taking the table of the predetermined format as an Excel table as an example, when the user sends a detection instruction to the detection system through the client, the relevant information of the content to be detected is organized into an Excel table; after the detection system detects the content to be detected, the detection result is filled in the Excel table, and the Excel table filled with the detection result is returned to the client, so that the user can view the detection result on the client. For example, after receiving the detection result fed back by the multimodal large model, the text detection algorithm can be used to cut the text of the detection result to obtain at least one of the detection score and the scoring reason; then call the Python graph processing (PIL, Python Imaging Library) library, and write at least one of the detection score and the scoring reason for each content to be detected into the Excel table submitted by the user to obtain the changed table. The changed table contains the relevant information of the content to be detected filled in by the user when sending the detection instruction, and the detection result obtained after the detection (such as at least one of the detection score and the scoring reason), and this information organization information is convenient for users to read, extract and save.

[0090] To summarize the above, the detection system and method proposed in the embodiments of the present disclosure use the FastAPI asynchronous framework to integrate multiple detection capabilities of a multimodal large model into the system. The business party can organize the relevant information of the content to be detected into an Excel table. Taking the text and image data quality assessment scenario as an example, the business party organizes the prompt and other information of the pictures and images to be reviewed into an Excel table. The detection system supports the following capabilities:

[0091] Read the table content according to the specified path parameters and extract the image information;

[0092] Define prompt templates according to the detection scenario to generate the corresponding model input format;

[0093] Call the multimodal large model, process the test results returned by the multimodal large model, and extract fields such as the test score and scoring reason in the test results;

[0094] The extracted fields such as test scores and scoring reasons are automatically filled back into the Excel spreadsheet to complete the fully automated testing process.

[0095] In some embodiments, the web application framework uses the Python Regular (Re) library to slice text fields to obtain the image quality assessment standard score and the reason for the score. Finally, the Python Imaging Library (PIL) library is used to write the data returned by the multimodal large model into an Excel spreadsheet to achieve a 100% automated image quality assessment function.

[0096] Figure 3 is a schematic diagram of an implementation framework of a detection method according to an embodiment of the present disclosure, Figure 3 In the example shown, the Web application framework is specifically a FastAPI service module, and the multimodal big model includes an open source multimodal big model and a closed source multimodal big model.

[0097] like Figure 3 As shown, the user (ie, the business party) uploads the content to be detected through the client 310, and the relevant data in the content to be detected is organized into a table, which is stored in the local service specified path. For example, the table includes the picture to be detected, and the prompt used by other large models (such as the Wensheng graph model) when generating the picture. After receiving the detection instruction from the client, the FastAPI service module 320 reads the table content, extracts the picture and text (such as prompt) content in the table, and searches for the corresponding scoring template prompt according to the detection type; for the extracted pictures, the FastAPI service module 320 can also perform format conversion on the extracted pictures, such as Base64 decoding. The FastAPI service module 320 generates a prompt for instructing the multimodal large model 330 to perform detection using the scoring template prompt, the extracted image (or the image after format conversion) and the text, and calls the multimodal large model 330, which performs detection based on the prompt to obtain the evaluation score and reason for evaluating the content to be detected, and returns the evaluation score and reason to the FastAPI service module 320. The FastAPI service module 320 receives the content returned by the multimodal large model 330, extracts the evaluation score and reason from the content, and fills the evaluation score and reason into the table provided by the client to perform table data processing, and then returns the processed table to the client 310.

[0098] In some implementations, the prompt template may include at least one of a task description, a scoring template, an output format, and an output example to define content such as scoring criteria and processing format.

[0099] The following is an example of a prompt template for AI text graph data quality assessment, including one or more of the following parts:

[0100] 1. Task description TASK_DESCRIPTION = ("This picture is generated by prompt: {}. Now you need to score the picture content based on this prompt and the input picture.\n"

[0101] "1. Understand the content of the image and accurately identify the subject and scene.\n"

[0102] "2. Compare the image content with its corresponding prompt to determine whether the image accurately describes and includes all the key information in the prompt. If the image does not fully express the content described by the prompt, the semantic relevance score should be low.\n"

[0103] "3. Determine whether the subject in the image matches the prompt description, including the subject's appearance, posture, and details. If the details are incomplete or do not match the prompt description, the subject effect score is low.\n"

[0104] "4. Evaluate whether the content in the image meets the expectations of the prompt, determine whether there are redundant, wrong or inappropriate elements, and ensure that the image is consistent with the prompt description. If there are irrelevant or inappropriate elements, reduce the image comfort score.\n"

[0105] "5. Check the overall composition and quality of the image, paying special attention to whether there are any problems such as misalignment, blur, or lack of clarity. Defects in composition and image quality will affect the overall quality score.\n"

[0106] "6. Evaluate the overall aesthetics of the image, including color, contrast, saturation, composition, light and shadow effects, etc., to ensure that the image is harmonious and attractive. If the light and shadow are not handled properly or the composition is confusing, the aesthetic score should be lowered.\n"

[0107] "Please note that the level of each rating dimension should reflect whether the image accurately and meticulously presents the prompt's requirements in all aspects while maintaining high quality, visual consistency, and aesthetics.\n" )

[0109] 2. Scoring template SCORES_TEMPLATE = (

[0110] "Each scoring dimension is scored based on a 1 to 5 score scale, supporting 1 decimal place.\n"

[0111] "Rating dimensions and standards:\n"

[0112] "Semantic relevance: The story's completeness is divided into five rating levels from 1 to 5 based on the completeness, correctness, and detailed description of the main elements of the picture.\n"

[0113] "*1 point: The content of the picture does not match the description in the prompt at all, and is missing an important subject.\n"

[0114] "*2 points: Part of the subject matches the prompt description, and the generated content is similar to the prompt but inaccurate and lacks completeness.\n"

[0115] "*3 points: The main body of the picture is complete and basically meets the description in the prompt, but some content in the prompt is missing, and details are missing.\n"

[0116] "*4 points: The main body of the picture is complete, consistent with the prompt description, including all the contents mentioned in the prompt, and lacks a few details.\n"

[0117] "*5 points: The main body and details of the picture are highly matched, exceeding the prompt description.\n"

[0118] "Subject effect: The subject effect is divided into five rating levels from 1 to 5 according to the subject's facial features, limbs and other details.\n"

[0119] "*1 point: The subject's facial features, fingers, etc. have obvious defects, which seriously affect the viewing experience.\n"

[0120] "*2 points: The facial features and limbs are partially overlapped, distorted or blurred, but the subject is still recognizable.

[0121] \n"

[0122] "*3 points: The facial features and limbs are basically intact, and do not affect the viewing experience. The picture is blurry and the subject is not clear.\n"

[0123] "*4 points: facial features and limbs are clear and in harmonious proportions, with no obvious defects and harmonious expressions.\n"

[0124] "*5 points: The facial features and limbs are delicate, the proportions are natural, and the expression matches the prompt description.\n"

[0125] "Rendering effect: The rendering effect is divided into five rating levels of 1-5 based on elements such as style and atmosphere.

[0126] \n"

[0127] "*1 point: The style is seriously inconsistent with the description in the prompt, and the color tone and lighting are inconsistent.\n"

[0128] "*2 points: The style basically meets the description in the prompt, but there is a strong conflict between the style and the subject, and the lighting and atmosphere of the picture are inconsistent.\n"

[0129] "*3 points: The style is consistent with the description in prompt, but the lighting is average, the color distribution is unreasonable, and the atmosphere effect is not ideal.\n"

[0130] "*4 points: The style and atmosphere are good in presenting the prompt requirements.\n"

[0131] "*5 points: The style and atmosphere perfectly meet the prompt requirements, and the atmosphere is full.\n"

[0132] "Image comfort: Based on factors such as fun, the image comfort is divided into five rating levels from 1 to 5\n"

[0133] "*1 point: There are obvious elements that make people uncomfortable, and the overall viewing experience is poor.\n"

[0134] "*2 points: There are a few uncomfortable elements that affect the viewing experience.\n"

[0135] "*3 points: No inappropriate elements, the overall picture is harmonious.\n"

[0136] "*4 points: The picture is harmonious and attractive.\n"

[0137] "*5 points: The picture is highly attractive and interesting.\n"

[0138] "Overall quality: The overall quality of the picture is divided into five rating levels from 1 to 5 based on the picture clarity and picture integrity\n"

[0139] "*1 point: The image is misplaced, blurred, the composition is chaotic, there is no logic, splicing, or missing.\n"

[0140] "*2 points: The image is partially misaligned and has poor clarity, and can be directly identified as a non-human work image.

[0141] \n"

[0142] "*3 points: The image clarity is basically acceptable, but there is some distortion.\n"

[0143] "*4 points: The image is clear, the composition is complete, and there are minor flaws.\n"

[0144] "*5 points: The image is highly clear and has no defects.\n"

[0145] "Aesthetics of the picture: Based on multiple dimensions such as composition beauty, color matching, picture layering, light and shadow effects, picture coordination, and visual appeal, the overall aesthetics is divided into five rating levels of 1-5.

[0146] \n"

[0147] "*1 point: unbalanced composition, disharmonious colors, poor sense of hierarchy, and unbalanced colors.\n"

[0148] "*2 points: poor composition, insufficient color and layering, stiff colors, obvious conflict between colors and elements, poor light and shadow and layering.\n"

[0149] "*3 points: reasonable composition, harmonious colors, acceptable layering and light and shadow effects, but lacks outstanding visual appeal.\n"

[0150] "*4 points: Good composition, harmonious color, layering, light and shadow treatment, and unified overall style.

[0151] \n"

[0152] "*5 points: The composition is exquisite, the color layers are rich, the light and shadow effects are perfect, and the picture is highly harmonious.

[0153] \n" )

[0155] 3. Format output ANSWER_TEMPLATE = ("Output format:\n"

[0156] "Please provide the rating and rating reason in Chinese JSON format, for example:\n"

[0157] "{'score':'Fill in the average score of each dimension here','details':{'Semantic relevance':'Fill in the specific score here','Fill in the specific reason here';'Main effect':'Fill in the specific score here','Fill in the specific reason here';'Rendering effect':'Fill in the specific score here','Fill in the specific reason here';'Picture comfort':'Fill in the specific score here','Fill in the specific reason here';'Overall quality':'Fill in the specific score here','Fill in the specific reason here';'Picture beauty':'Fill in the specific score here','Fill in the specific reason here'}}\n" )

[0159] 4. Sample output SAMPLE_OUTPUT = (

[0160] "Output example:\n"

[0161] "{'score':2.7,'details':{'Semantic relevance':'4.1 points. The main body of the picture is complete, basically consistent with the prompt description, lacking a few details.','Main body effect':'4.5 points. The facial features, fingers and limbs of the main body are clear and natural, consistent with the description.','Rendering effect':'3.0 points. The style and atmosphere do not match the prompt, and the lighting is slightly poor.','Picture comfort':'2.5 points. The picture is not attractive, there are uncomfortable elements, and the viewing experience is not good.','Overall quality':'1.3 points. The image clarity is low, slightly different from the prompt description effect.','Picture beauty':'2.7 points. The composition is not beautiful, and the layering and color matching are not coordinated.'}}" )

[0163] Using the above prompt template, the FastAPI service module generates a prompt; the multimodal large model detects the content to be detected based on the prompt, and returns the detection result to the FastAPI service module, which can be expressed in JSON format. After receiving the detection result, the FastAPI service module cuts the text field in the detection result to obtain information such as evaluation score and scoring reason, and fills the obtained evaluation score and scoring reason into an Excel table, and provides the Excel table to the user.

[0164] The following Table 1 is an example of an Excel table, which contains multiple rows of information, each row corresponding to an image to be detected. The first two columns of the Excel table contain the AI ​​text image input by the user and the prompt input when the text model generates the image; the last two columns of the Excel table contain the content extracted and filled into the Excel table according to the detection results input by the FastAPI service module.

[0165] Table 1

[0166]

[0167]

[0168] Taking the application scenario of badcase mining in algorithm scenarios of on-site projects as an example, in some implementations, the user fills in an Excel table through the client with pictures of sub-scenes with warning boxes to be detected, basic prompts entered when other large models are used to identify the warning content in the pictures, and related event definitions; the FastAPI service module reads the Excel table, generates a corresponding prompt based on the read content and the pre-set prompt template, and calls the multimodal large model; the multimodal large model evaluates the pictures of sub-scenes with warning boxes based on the prompt.

[0169] For example, the basic prompt entered by the user is "Based on the national standard definition of urban management events, warn about exposed garbage in the picture and mark it with a red frame", and also enters the warning picture marked by other large models based on the basic prompt. Figure 4 As shown. The detection system proposed in the embodiment of the present disclosure is used to detect whether the warning image is correct, that is, to detect whether the warning is a bad case. For example, if the detection result generated by the detection system proposed in the embodiment of the present disclosure is "there is exposed garbage in the red box in the picture", then the warning image is correct, that is, the warning is not a bad case.

[0170] In some implementations, the multimodal large model proposed in the embodiments of the present disclosure supports multi-round conversations, can achieve strategy optimization through parameter pass-through, and further improve the recognition accuracy of the multi-modal large model for complex scene images.

[0171] Taking image annotation evaluation as an example, a multimodal large model is used to detect the annotations of the image annotation team. For example, the user organizes the images to be evaluated into an Excel table, and the FastAPI service module reads the Excel table and extracts the images to be evaluated in the Excel table. According to the preset prompt template, a multi-round dialogue prompt is generated. The following is an example of a multi-round prompt design and the data returned by the multimodal large model, including:

[0172] The first round of prompts: You need to determine whether there is anyone in the picture who is using a mobile phone or making a call. Is there anyone in the picture? If so, what are these people doing?

[0173] Model first round return data: In this picture, I see a person using a mobile phone. He is standing in the store with the phone in his hand and appears to be making a call. In addition, there are other people in the store, but they don't have mobile phones or make calls. These people may be browsing products, shopping, or communicating with store employees.

[0174] Second round of prompts: Is it possible that these people use mobile phones to do what they are doing?

[0175] The second round of model return data: These people may be browsing products, shopping, or communicating with store clerks. These people may use mobile phones for what they are doing, but in this picture, only that person is using a mobile phone.

[0176] Third round prompt: Your answer must be in Chinese and output in JSON code format, for example: {'phone':1,'reason':'There is a person using a mobile phone or making a phone call in the picture'} (0 means no behavior is recognized; 1 means the use of a mobile phone or making a phone call is recognized)

[0177] The third round of model return data:

[0178] #1 point, reason: There is a person using a mobile phone or making a call in the picture, the re-judgment result is accurate

[0179] json\n{\n\"phone\":1,\n\"reason\":\"There is someone using a mobile phone or making a call in the picture\"\n}

[0180] After the multimodal large model returns the above detection results, the FastAPI service module organizes the detection results, for example, into text information for describing standard information, fills the text information into an Excel table, and returns the Excel table filled with text information to the client.

[0181] In summary, it can be seen that the detection method provided by the embodiment of the present disclosure can integrate the capabilities of a multimodal large model, realize the functions of quality assessment of AI text-generated graph data, mining badcases of algorithm scenarios in project sites, and evaluating images annotated by the image annotation team. Integrating a multimodal large model on the engineering side can realize a fully automated process for image quality assessment, reduce manual intervention, and improve work efficiency and assessment quality. Automatic quality assessment through a multimodal large model greatly reduces the workload and time of manual review, improves efficiency, and reduces costs. In addition, the use of a multimodal large model can reduce the assessment bias caused by human factors and improve the objectivity and accuracy of the assessment through machine learning and self-training models. The multimodal large model can process a large amount of image data, is not limited by the amount of data for manual review, and can cope with large-scale image quality assessment tasks. In addition, the multimodal large model can continuously improve the accuracy of image quality assessment through continuous learning and optimization, and adapt to the ever-changing image quality standards.

[0182] The disclosed embodiment also proposes a detection system based on a large model. Figure 5 is a schematic diagram of the structure of a large model-based detection system 500 according to an embodiment of the present disclosure, including a Web application framework and a multimodal large model, wherein:

[0183] The Web application framework 510 is used to receive a detection instruction from a client, the detection instruction including content to be detected; determine a detection type based on the detection instruction, and call a multimodal large model based on the detection type; and format the detection result returned by the multimodal large model, and return the formatted detection result to the client;

[0184] The multimodal large model 520 is used to detect the content to be detected based on the call and return the detection result to the Web application framework.

[0185] In some embodiments, the detection type includes at least one of artificial intelligence (AI) text image data quality assessment, algorithm scene error mining, and image annotation assessment.

[0186] In some implementations, the multimodal big model 520 includes at least two application interfaces, each application interface corresponding to a detection capability of a detection type.

[0187] In some implementations, the Web application framework 510 is used to call an application interface based on the detection type to implement the call to the multimodal large model;

[0188] The application interface called by the Web application framework 510 corresponds to the detection capability of the detection type.

[0189] In some implementations, the detection instruction includes a table in a predetermined format, and the table in the predetermined format includes a plurality of contents to be detected.

[0190] In some implementations, the web application framework 510 is used to:

[0191] Extract a table in a predetermined format;

[0192] Extracting content of a specified column from the table in the predetermined format according to a predetermined strategy, wherein the content of the specified column includes an image and / or text;

[0193] The extracted image and / or text content is stored in a local service path, and the extracted image and / or text content is converted into a format.

[0194] In some implementations, the web application framework 510 is used to:

[0195] According to the detection type, obtain the pre-saved prompt word template corresponding to the detection type;

[0196] Generate prompt words for instructing the multimodal large model 520 to perform detection according to the prompt word template and the extracted content;

[0197] The multimodal large model 520 is called, and the generated prompt word is sent to the multimodal large model 520 .

[0198] In some embodiments, the multimodal macromodel 520 is used to:

[0199] According to the received prompt words, each piece of content to be detected is detected respectively;

[0200] According to at least one of the output format and the output example required by the prompt word, the detection result of each to-be-detected content is returned to the Web application framework.

[0201] In some implementations, the web application framework 510 is used to:

[0202] Performing text segmentation on the detection results of each piece of content to be detected to obtain at least one of a detection score and a scoring reason;

[0203] Writing at least one of the test score and the scoring reason for each content to be tested into the table of the predetermined format to obtain a table filled with the test results;

[0204] Return the form filled with the test results to the client.

[0205] For the description of specific functions and examples of each module and submodule of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0206] In the technical solution disclosed in the present invention, the acquisition, storage and application of personal information of users involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0207] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0208] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0209] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0210] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0211] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the detection method in any other appropriate manner (e.g., by means of firmware).

[0212] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0213] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0214] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0215] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0216] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0217] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0218] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0219] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A detection method based on a large model, comprising: Receiving a detection instruction, wherein the detection instruction includes content to be detected; determining a detection type based on the detection indication, and invoking a multimodal large model based on the detection type; Using the multimodal large model to detect the content to be detected based on the call; Receiving the detection result returned by the multimodal large model; The detection results are formatted and the formatted detection results are sent.

2. The method according to claim 1, wherein: The detection type includes at least one of artificial intelligence (AI) text image data quality assessment, algorithm scene error mining, and image annotation assessment.

3. The method according to claim 1 or 2, before receiving the detection indication, further comprising: Pre-training and deploying a multi-modal large model that provides detection capabilities for at least two of the detection types; At least two application interfaces are established, each of which corresponds to a detection capability of the detection type.

4. The method according to claim 3, wherein: The calling of the corresponding capability of the multimodal large model based on the detection type includes: Calling an application interface based on the detection type to implement calling the multimodal large model; The called application interface corresponds to the detection capability of the detection type.

5. The method according to any one of claims 1 to 4, wherein: The detection instruction includes a table in a predetermined format, and the table in the predetermined format includes a plurality of contents to be detected.

6. The method according to claim 5, wherein: Before determining the detection type based on the detection indication, the method further includes: extracting the table in the predetermined format; Extracting content of a designated column from the table in the predetermined format according to a predetermined strategy, wherein the content of the designated column includes an image and / or text; The extracted image and / or text content is stored in a local service path, and the extracted image and / or text content is converted into a format.

7. The method according to claim 6, wherein: The calling of the multimodal large model based on the detection type includes: According to the detection type, obtaining a pre-saved prompt word template corresponding to the detection type; Generating prompt words for instructing the multimodal large model to perform detection according to the prompt word template and the extracted content; The multimodal large model is called, and the generated prompt word is sent to the multimodal large model.

8. The method according to claim 7, wherein: The detecting the content to be detected by using the multimodal large model based on the calling includes: Using the multimodal large model to detect each of the to-be-detected contents according to the received prompt words; The multimodal large model is used to determine the detection result of each of the to-be-detected contents according to at least one of the output format and the output example required by the prompt word.

9. The method according to claim 8, wherein: The formatting of the test results and sending the formatted test results includes: Performing text segmentation on the detection results of each of the to-be-detected contents respectively to obtain at least one of a detection score and a scoring reason; Writing at least one of the detection score and the scoring reason for each of the to-be-detected contents into the table of the predetermined format to obtain a table filled with the detection results; Send the form filled in with the test results.

10. A large model-based detection system, comprising a Web application framework and a multimodal large model, wherein: The Web application framework is used to receive a detection instruction from a client, wherein the detection instruction includes content to be detected; determine a detection type based on the detection instruction, and call the multimodal large model based on the detection type; It is also used to format the detection results returned by the multimodal large model and return the formatted detection results to the client; The multimodal large model is used to detect the content to be detected based on the call and return the detection result to the Web application framework.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

13. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images