A detection method, system, device, and storage medium based on a large model.

Abstract

This disclosure provides a detection method, system, device, and storage medium based on a large model, relating to the field of computer technology, particularly data processing and artificial intelligence. The specific implementation scheme includes: receiving a detection instruction containing the content to be detected; determining the detection type based on the detection instruction and invoking a multimodal large model based on the detection type; using the multimodal large model to detect the content to be detected based on the invocation; receiving the detection result returned by the multimodal large model; formatting the detection result; and sending the formatted detection result. This disclosure can provide automated detection for multiple types based on a unified detection platform.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of data processing and artificial intelligence. Background Technology

[0002] With the rapid development of artificial intelligence (AI), especially the emergence of large language models and large image generation models, more and more business sectors are beginning to integrate these advanced AI capabilities. These large models can typically handle complex tasks, such as image recognition and text-to-image generation.

[0003] However, despite the powerful capabilities of large models, the data they generate is not always entirely as expected and may contain quality issues such as inaccuracies, poor visual effects, or lack of creativity. Therefore, it is necessary to inspect the data generated by large models and / or the data labeled by the annotation team to enable data filtering, evaluation, and scoring. Summary of the Invention

[0004] This disclosure provides a detection method, system, device, and storage medium based on a large model.

[0005] According to one aspect of this disclosure, a detection method based on a large model is provided, comprising:

[0006] Receive a detection instruction, which contains the content to be detected;

[0007] The detection type is determined based on the detection indication, and the multimodal large model is invoked based on the detection type;

[0008] This multimodal large model is used to detect the content to be detected based on the call;

[0009] Receive detection results returned by a large multimodal model;

[0010] The test results are formatted and then sent.

[0011] According to another aspect of this disclosure, a detection system based on a large model is provided, comprising a web application framework and a multimodal large model, wherein,

[0012] The web application framework is used to receive detection instructions from the client, which contain the content to be detected; determine the detection type based on the detection instructions, and call the multimodal large model based on the detection type; it is also used to format the detection results returned by the multimodal large model and return the formatted detection results to the client.

[0013] This multimodal large model is used to perform detection on the content to be detected based on the call, and return the detection results to the web application framework.

[0014] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0015] At least one processor; and

[0016] The memory is communicatively connected to the at least one processor; wherein,

[0017] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform any of the methods described in the present disclosure.

[0018] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0019] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0020] This disclosure proposes a detection method based on a large model. The method receives a detection instruction, determines the detection type based on the instruction, and then invokes a multimodal large model to perform the corresponding type of detection. After receiving the detection results from the large model, the method can also format the results and provide the formatted results back to the user. In this way, automated content detection based on a multimodal large model can be achieved, and a unified detection platform can be provided externally.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0022] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0023] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure;

[0024] Figure 2 This is a flowchart illustrating the implementation of a detection method according to an embodiment of the present disclosure;

[0025] Figure 3 This is a schematic diagram illustrating the implementation framework of a detection method according to an embodiment of the present disclosure;

[0026] Figure 4 These are alarm images according to an embodiment of this disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of a large-model-based detection system 500 according to an embodiment of the present disclosure;

[0028] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0029] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0030] The term "and / or" in this disclosure indicates that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. The term "at least one" in this document means any combination of at least two of a plurality of options, such as including at least one of A, B, and C, which can mean including any one or more elements selected from the set of A, B, and C. The terms "first" and "second" in this document refer to and distinguish multiple similar technical terms, and do not imply a specific order or a limitation to only two. For example, "first feature" and "second feature" refer to two types / two features; the first feature can be one or more, and the second feature can also be one or more.

[0031] With the rapid development of artificial intelligence, especially the emergence of large-scale language models and large-scale image generation models, more and more business sectors are beginning to integrate these advanced AI capabilities. These large models can typically handle complex tasks, such as image recognition and text-to-image generation.

[0032] In image generation scenarios, these models can generate corresponding images based on text descriptions, or perform operations such as image style transfer and image inpainting. The generated images can be used in various applications, such as intelligent agent content, smart cities, smart transportation, and smart industry.

[0033] However, despite the power of large models, the data they generate is not always entirely as expected and may contain quality issues such as inaccuracies, poor visual effects, and lack of creativity. Therefore, it is necessary to inspect the data generated by large models to assess the quality of the data produced by large modules.

[0034] In existing technologies, manual review is commonly used. Manual review significantly increases labor costs, reduces evaluation efficiency, and results in lower evaluation quality.

[0035] For example, in the quality assessment of AI-generated text images, each image generated by the large text-generating model needs to be manually reviewed, and scored based on the degree of matching between the prompt and the generated image to evaluate image quality. Manual review of AI images generated by text-generating models is not only time-consuming and labor-intensive, but also carries the risk of inconsistent evaluation results due to differences in individual standards and understanding. Furthermore, prolonged review may lead to fatigue and distraction, further affecting the accuracy of the assessment.

[0036] For example, in the field project algorithm scenario badcase mining (referred to as algorithm scenario badcase mining), alarms generated by real cameras using online algorithms need to be manually reviewed one by one to determine whether the algorithm scenario definition corresponding to the alarm data can be defined as a badcase. This method is labor-intensive, and manual evaluation can easily become tiring when the data volume is large. In addition, it takes a long time to become familiar with the definitions of more than a hundred algorithm scenarios.

[0037] For example, in scenarios where image annotation teams review image annotations, after data annotators complete their annotations, these annotations need to be manually reviewed to verify their accuracy. This manual review process is costly, and the results can be influenced by individual subjective factors, leading to inconsistencies in the annotations.

[0038] In summary, all the methods described above involve manual observation and review. The main drawbacks of manual methods are low efficiency, high cost, and strong subjectivity. Currently, there is no unified detection platform available for multiple detection types to achieve automated detection across various scenarios.

[0039] Based on the above, this disclosure proposes a detection system and method that can perform various types of detection based on the capabilities of a multimodal large model, thereby achieving automated detection, reducing manual review costs, and improving detection accuracy.

[0040] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this disclosure, such as... Figure 1As shown in the illustration, the application scenario diagram of this disclosure may include, but is not limited to, a client 110 and a detection system 120. The client 110 and the detection system 120 can communicate through any type of wired or wireless network. Specifically, the client 110 can be used to receive a user's detection instruction and send the user's detection instruction to the detection system 120; the detection system 120 can receive the detection instruction sent by the client, perform detection on the content to be detected based on the detection instruction, and return the detection results to the client 110 after processing them. The client 110 proposed in this disclosure includes, but is not limited to, electronic devices such as mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, game consoles, e-book readers, multimedia playback devices, and wearable devices; the detection system 120 may include electronic devices or servers for providing automated detection services to the client 110. Furthermore, this disclosure does not impose a specific limitation on the number of behavioral clients 110. For example, the application scenario diagram of this disclosure may include one or more clients 110.

[0041] This disclosure presents a detection method based on a large model. Figure 2 This is a flowchart illustrating the implementation of a large-model-based detection method according to an embodiment of the present disclosure, including:

[0042] S210. Receive a detection instruction, which contains the content to be detected;

[0043] S220. Determine the detection type based on the detection indication, and call the multimodal large model based on the detection type;

[0044] S230. Using the multimodal large model based on the aforementioned call, detect the content to be detected;

[0045] S240. Receive the detection results returned by the multimodal large model;

[0046] S250. Format the test result and send the formatted test result.

[0047] This embodiment of the disclosure can receive detection instructions sent by users from clients, determine the detection type based on the detection instructions, and call a multimodal large model to perform the corresponding type of detection based on the detection type. After receiving the detection results from the large model, the detection results can also be formatted and the formatted detection results can be fed back to the client. In this way, automated content detection based on a multimodal large model can be realized, and a unified detection platform can be provided to external parties.

[0048] In some implementations, embodiments of this disclosure can be applied to detection systems including web application frameworks and multimodal large models. When the detection result is used to implement the detection method of this solution, the detection method may specifically include:

[0049] The web application framework receives a detection instruction from the client, which contains the content to be detected.

[0050] The web application framework determines the detection type based on the detection indication and calls the multimodal large model based on the detection type;

[0051] Based on this call, the multimodal large model performs detection on the content to be detected and returns the detection results to the web application framework;

[0052] The web application framework formats the detection results and returns the formatted results to the client.

[0053] The above example constructs a detection system that includes a web application framework and a multimodal large model, and can be used to perform various types of detection.

[0054] In some implementations, the web application framework includes FastAPI, a high-performance web framework that provides the ability to quickly build APIs in a simple way. FastAPI supports asynchronous programming, making it easier to handle a large number of concurrent requests.

[0055] In some implementations, the detection types include at least one of AI-generated image data quality assessment, algorithm scene error mining, and image annotation evaluation. Therefore, the detection method proposed in this disclosure can provide multiple types of automated detection based on a unified detection platform, such as quality assessment of images generated by AI-generated image models, mining of error cases in alarm data detected by other large models in on-site project algorithm scenarios, and evaluation of data annotated by image annotation teams.

[0056] In some implementations, embodiments of this disclosure pre-train and deploy a multimodal large model capable of providing detection capabilities for at least two of the above-described detection types;

[0057] Establish at least two application interfaces (such as API interfaces), with each application interface corresponding to a detection capability of a detection type.

[0058] For example, embodiments of this disclosure pre-train a multimodal large model, which is capable of providing AI-generated image data quality assessment, algorithm scene error detection, and image annotation evaluation; and establishes three API interfaces, each corresponding to the aforementioned three detection capabilities. Furthermore, embodiments of this disclosure deploy this multimodal large model, such as deploying it on a graphics card device.

[0059] This API interface enables web application frameworks to call multimodal large models. In one example, the web application framework calls the application interface based on the detection type to access the multimodal large model; the application interface called by the web application framework corresponds to the detection capability of that detection type. By establishing corresponding application interfaces for different detection capabilities of the multimodal large model, the web application framework can easily call different capabilities of the multimodal large model, thereby simplifying the model invocation process.

[0060] In some examples, an open-source multimodal large model is used, and this model is trained and deployed offline. For instance, data from business-related algorithm scenarios is used for data cleaning, labeling, self-training, and model parameter tuning. These operations improve the model's recognition accuracy in specific scenarios.

[0061] Furthermore, based on AI-generated images provided by the intelligent content business line, bad case images of algorithmic scenarios, and image data from the data annotation team, targeted prompts can be made to multi-model and large-scale models. This allows for precise optimization of the understanding capabilities of multi-model and large-scale models for different needs, thereby achieving better recognition results.

[0062] In some implementations, step S220 may further include:

[0063] Extract tables with a predefined format;

[0064] According to a predetermined strategy, extract the content of a specified column from a table with a predetermined format, the content of which includes images and / or text;

[0065] The extracted images and / or text content are stored in the local service path, and the format of the extracted images and / or text content is converted.

[0066] The above process extracts images and / or text related to the detection content, providing conditions for subsequent use of multimodal large models for detection.

[0067] In some implementations, utilizing the FastAPI asynchronous framework, embodiments of this disclosure can integrate multiple detection capabilities of multimodal large models into the system. When a user sends content to be detected through a client, data upload can be supported according to a pre-formatted table. For example, the detection instruction sent by the user through the client includes a pre-formatted table containing multiple pieces of content to be detected. Taking AI text-based image data quality assessment as an example, the user (i.e., the business party) can organize information such as the image to be reviewed and the prompt used by the text-based image model when generating the image into a pre-formatted table (such as an Excel spreadsheet). By setting a pre-formatted table, it is convenient for users to upload content to be detected and for the web application framework to perform operations such as saving and retrieving the content to be detected.

[0068] Accordingly, the web application framework extracts the content to be detected, including:

[0069] The web application framework extracts the table in the predefined format;

[0070] The web application framework extracts the content of specified columns from a table with a predefined format according to a predefined strategy. The content of the specified columns includes images and / or text.

[0071] The web application framework stores the extracted images and / or text content in a local service path and performs format conversion on the extracted images and / or text content to prepare for calling the multimodal large model.

[0072] By setting predefined table formats and strategies, web application frameworks can easily extract content for detection from the tables, improving the speed and accuracy of information extraction.

[0073] In some implementations, corresponding prompt templates can be pre-generated based on different detection types. The content of these prompt templates is related to the detection content, scoring criteria, output result format, and other information corresponding to the detection type. This embodiment of the present disclosure pre-generates and saves prompt templates for different detection types. When calling the multimodal large model, a corresponding prompt can be generated based on the prompt template corresponding to the current detection type and the extracted content. The generated prompt is then used to call the multimodal large model to perform the detection task.

[0074] Taking the use of a web application framework to call a multimodal large model as an example, when calling a multimodal large model, the web application framework can generate a corresponding prompt according to the detection type and the content to be detected, and input the prompt into the multimodal large model, which will then perform detection and evaluation based on the prompt.

[0075] In some examples, the FastAPI service uses the openpyxl and pandas libraries to extract all content from an Excel spreadsheet, and extracts images and text content from specified columns according to a strategy. All images are stored in the local service path, and the extracted images are converted to base64 format in preparation for calling open-source multimodal large models.

[0076] In some implementations, a multimodal large model is invoked based on the detection type, including:

[0077] The web application framework retrieves the pre-saved prompt template corresponding to the detection type based on the detection type.

[0078] The web application framework generates prompts based on the prompt template and extracted content to instruct multiple models and large models to perform detection.

[0079] The web application framework calls the multimodal big model and sends the generated prompts to the multimodal big model.

[0080] By pre-setting prompt templates corresponding to various detection types, and generating prompts matching the detection type based on the prompt templates when calling multimodal large models in the web application framework, the prompts required for multimodal large model detection can be generated quickly, and the generated prompts are uniform and standardized.

[0081] For example, a web application framework can generate multiple prompt words based on prompt word templates and multiple extracted content to be detected, which are used to instruct a multi-model large model to perform detection. Each prompt word is used to instruct the multi-model large model to perform detection on a single content to be detected.

[0082] The multimodal big model detects each piece of content to be detected based on the received prompt words; for example, a web application framework generates multiple prompt words for multiple pieces of content to be detected and sends these multiple prompt words to the multimodal big model.

[0083] After detection, the multimodal large model returns the detection results of each piece of content to be detected to the web application framework according to at least one of the output formats and output examples required by each prompt word. The returned detection results can be in JSON format, which meets the output format and / or output example required in the prompt.

[0084] After receiving the detection results of multiple objects to be detected from the multimodal large model, the web application framework formats the detection results and returns the formatted detection results to the client. For example, these may include:

[0085] The web application framework performs text segmentation on the detection results of each piece of content to be detected in order to obtain at least one of the detection score and scoring reasons;

[0086] The web application framework will write at least one of the detection scores and scoring reasons for each piece of content to be detected into a table in a predefined format to obtain a table with the detection results filled in.

[0087] The web application framework returns a form to the client after the test results have been filled in.

[0088] By using a web application framework to convert the detection results returned by the large model into a unified format and write the relevant information into a table, data transmission and user viewing can be facilitated.

[0089] Taking an Excel spreadsheet as an example, when a user sends a detection command to the detection system through a client, the relevant information of the content to be detected is organized into an Excel spreadsheet. After the detection system detects the content, it fills the detection results into the Excel spreadsheet and returns the updated spreadsheet to the client for the user to view. For instance, after receiving the detection results from a multimodal large model, a text detection algorithm can be used to segment the results to obtain at least one of the detection score and the reason for the score. Then, the Python Imaging Library (PIL) is called to write at least one of the detection scores and the reason for the score for each piece of content to be detected into the user-submitted Excel spreadsheet, resulting in a modified spreadsheet. This modified spreadsheet contains the relevant information of the content to be detected that the user filled in when sending the detection command, as well as the detection results obtained after detection (such as at least one of the detection score and the reason for the score). This information organization makes it easy for users to read, extract, and save.

[0090] In summary, the detection system and method proposed in this disclosure utilize the FastAPI asynchronous framework to integrate multiple detection capabilities of a multimodal large model into the system. Business users can organize relevant information about the content to be detected into an Excel spreadsheet. Taking the text image data quality assessment scenario as an example, the business user organizes information such as the images to be reviewed and image prompts into an Excel spreadsheet. The detection system supports the following capabilities:

[0091] Read table contents and extract image information based on specified path parameters;

[0092] Define a prompt template based on the detection scenario to generate the corresponding model input format;

[0093] Call the multimodal large model, process the detection results returned by the multimodal large model, and extract fields such as detection score and scoring reason from the detection results;

[0094] The extracted detection scores and scoring reasons are automatically populated into an Excel spreadsheet, completing the fully automated detection process.

[0095] In some implementations, the web application framework uses the Python Regular Expression (Re) library to segment text fields to obtain image quality assessment scores and the reasons for those scores. Finally, the Python ImagingLibrary (PIL) library is used to write the data returned by the multimodal large model into an Excel spreadsheet, achieving a 100% automated image quality assessment function.

[0096] Figure 3 This is a schematic diagram illustrating the implementation framework of a detection method according to an embodiment of the present disclosure. Figure 3 In the example shown, the web application framework is specifically the FastAPI service module, and the multimodal big model includes both open-source and closed-source multimodal big models.

[0097] like Figure 3 As shown, the user (i.e., the business party) uploads the content to be detected through client 310. The relevant data in the content to be detected is organized into a table, which is stored in a local service-specified path. For example, the table includes the image to be detected, as well as the prompts used by other large models (such as the text-to-image model) when generating the image. After receiving the detection instruction from the client, the FastAPI service module 320 reads the table content, extracts the image and text (such as the prompt) content from the table, and finds the corresponding scoring template prompt according to the detection type. For the extracted image, the FastAPI service module 320 can also perform format conversion, such as Base64 decoding. The FastAPI service module 320 uses a scoring template prompt, extracted images (or images after format conversion), and text to generate a prompt instructing the multimodal large model 330 to perform detection. It then calls upon the multimodal large model 330 to perform detection based on the prompt, obtaining an evaluation score and reason for the content to be detected. The evaluation score and reason are then returned to the FastAPI service module 320. Upon receiving the content returned by the multimodal large model 330, the FastAPI service module 320 extracts the evaluation score and reason from it, fills the score and reason into a table provided by the client for data processing, and then returns the processed table to the client 310.

[0098] In some implementations, the prompt template may include at least one of the following: task description, scoring template, output format, and output example, to define scoring criteria and processing format, etc.

[0099] Below is an example of a prompt template for AI-generated image data quality assessment, including one or more of the following sections:

[0100] 1. Task Description: TASK_DESCRIPTION = ("This image was generated using the prompt: {}. Now you need to score the content of the image based on this prompt and the input image.\n"

[0101] 1. Understand the content of the image and accurately identify the subject and scene.

[0102] 2. Compare the image content with its corresponding prompt to determine whether the image accurately describes and includes all the key information in the prompt. If the image fails to fully express the content described in the prompt, the semantic relevance score should be low.

[0103] 3. Determine if the subject in the image matches the prompt description, including the subject's appearance, pose, and details. If the details are incomplete or do not match the prompt description, the subject's performance score will be low.

[0104] 4. Evaluate whether the content in the image matches the expectations of the prompt, determine if there are any redundant, incorrect, or inappropriate elements, and ensure that the visuals are consistent with the prompt description. If irrelevant or uncomfortable elements are found, lower the visual comfort score.

[0105] 5. Check the overall composition and quality of the image, paying particular attention to any misalignment, blurriness, or insufficient sharpness. Defects in composition and image quality will affect the overall quality score.

[0106] 6. Evaluate the overall aesthetics of the image, including color, contrast, saturation, composition, and lighting effects, ensuring the image is harmonious and appealing. If the lighting is poorly handled or the composition is chaotic, the aesthetic score should be lowered.

[0107] Please note that the score for each dimension should reflect whether the image accurately and meticulously presents the prompt's requirements in all aspects, while maintaining high quality, visual consistency, and aesthetic appeal. )

[0109] 2. Scoring template SCORES_TEMPLATE = (

[0110] Each rating dimension is scored on a scale of 1 to 5, with one decimal place supported.

[0111] Scoring Dimensions and Standards:\n

[0112] "Semantic relevance: Story completeness is rated on a scale of 1-5 based on elements such as the integrity and accuracy of the main subject of the image, and the detail in the descriptions."

[0113] *1 point: The content of the video does not match the prompt description at all and lacks an important subject.\n

[0114] *2 points: Part of the main content matches the prompt description, but the generated content is similar to the prompt but inaccurate and lacks completeness.\n

[0115] *3 points: The main subject of the image is complete and basically matches the prompt description, but some content from the prompt is missing, and details are lacking.\n

[0116] *4 points: The main subject of the image is complete, conforms to the prompt description, and includes all the content mentioned in the prompt, but lacks a few details.\n

[0117] *5 points: The subject and details in the image are highly matched, exceeding the prompt description.\n

[0118] "Main subject effect: Based on the details of the subject's facial features, limbs, etc., the main subject effect is divided into five rating levels from 1 to 5."

[0119] *1 point: The subject's facial features, fingers, etc., have obvious defects, severely affecting the viewing experience.\n

[0120] *2 points: Facial features and limbs are partially overlapping, distorted or blurred, but the main body is still recognizable.

[0121] \n"

[0122] *3 points: Facial features and limbs are basically complete and do not affect the overall appearance; however, the image is blurry and the subject is unclear.

[0123] *4 points: Facial features and limbs are clear and proportionally harmonious, with no obvious defects, and the expression is natural.

[0124] *5 points: Facial features and limbs are detailed and proportionate, and the expression matches the prompt description.\n

[0125] Rendering effect: The rendering effect is divided into five rating levels from 1 to 5 based on elements such as style and atmosphere.

[0126] \n"

[0127] *1 point: The style is seriously inconsistent with the prompt description; the color tone and lighting are jarring.\n

[0128] *2 points: The style generally matches the prompt description, but there is a strong conflict between the style and the subject matter; the lighting and atmosphere in the image are also inconsistent.\n

[0129] *3 points: The style matches the prompt description, but the lighting is mediocre, the color distribution is unreasonable, and the overall atmosphere is unsatisfactory.\n

[0130] *4 points: The style and atmosphere effectively conveyed the prompt requirements.\n

[0131] *5 points: The style and atmosphere perfectly meet the requirements of the prompt, creating a strong sense of atmosphere.\n

[0132] Visual comfort level: Based on elements such as entertainment value, visual comfort is rated on a scale of 1 to 5.

[0133] *1 point: Contains elements that are noticeably disturbing; overall impression is poor.\n

[0134] *2 points: Contains a few inappropriate elements, affecting the viewing experience.\n

[0135] *3 points: No offensive elements, overall harmonious visuals.\n

[0136] *4 points: The visuals are harmonious and have a certain appeal.\n

[0137] *5 stars: The visuals are highly engaging and full of fun.\n

[0138] Overall Quality: Based on image sharpness and image integrity, the overall image quality is rated on a scale of 1 to 5.

[0139] *1 point: Image misalignment, blurriness, chaotic composition, illogical, spliced, or missing elements.\n

[0140] *2 points: The image is partially misaligned and has poor clarity, and can be directly identified as a non-human-made image.

[0141] \n"

[0142] *3 points: Image clarity is generally acceptable, but there is some distortion.\n

[0143] *4 points: The image is clear, the composition is complete, and there are minor flaws.\n

[0144] *5 stars: The image is highly clear and free of any flaws.\n

[0145] "Visual appeal: Based on multiple dimensions such as composition aesthetics, color matching, sense of layering, lighting effects, visual harmony, and visual attractiveness, the overall visual appeal is divided into five rating levels from 1 to 5.

[0146] \n"

[0147] *1 point: Unbalanced composition, disharmonious colors, poor sense of depth, and unbalanced colors.\n

[0148] *2 points: Poor composition, insufficient color and depth, harsh colors, obvious clashes between colors and elements, and poor light and shadow and depth.

[0149] *3 points: The composition is reasonable, the colors are harmonious, and the sense of layering and lighting effects are acceptable, but it lacks outstanding visual appeal.

[0150] *4 points: The composition is good, the colors, layers and lighting are well-coordinated, and the overall style is consistent.

[0151] \n"

[0152] *5 points: The composition is exquisite, the color layers are rich, the lighting effects are perfect, and the picture is highly harmonious.

[0153] \n" )

[0155] 3. Formatted output ANSWER_TEMPLATE = ("Output format:\n"

[0156] Please provide your rating and reason for rating in Chinese JSON format, for example: \n

[0157] {'score':'Average score for each dimension','details':{'Semantic relevance':'Specific score','Specific reason'; 'Main effect':'Specific score','Specific reason'; 'Rendering effect':'Specific score','Specific reason'; 'Visual comfort':'Specific score','Specific reason'; 'Overall quality':'Specific score','Specific reason'; 'Visual aesthetics':'Specific score','Specific reason'}}\n )

[0159] 4. Example output: SAMPLE_OUTPUT = (

[0160] Output example:\n

[0161] "{'score': 2.7,'details': {'Semantic relevance': '4.1 points. The main subject is complete and largely consistent with the prompt description, but lacks a few details.','Main subject effect': '4.5 points. The facial features, fingers, and limbs are clear and natural, consistent with the description.','Rendering effect': '3.0 points. The style and atmosphere do not match the prompt, and the lighting is slightly inadequate.','Visual comfort': '2.5 points. The image is unattractive, contains unpleasant elements, and has a poor viewing experience.','Overall quality': '1.3 points. The image clarity is low, slightly different from the prompt description.','Visual aesthetics': '2.7 points. The composition is unattractive, and the sense of layering and color matching are not harmonious.'}}" )

[0163] Using the aforementioned prompt template, the FastAPI service module generates a prompt. The multimodal large model performs detection on the content to be detected based on this prompt and returns the detection results to the FastAPI service module. These results can be represented in JSON format. After receiving the detection results, the FastAPI service module segments the text fields to obtain information such as evaluation scores and reasons for scoring. It then fills the obtained evaluation scores and reasons for scoring into an Excel spreadsheet and provides this Excel spreadsheet to the user.

[0164] Table 1 below is an example of an Excel spreadsheet. The Excel spreadsheet contains multiple rows of information, each corresponding to an image to be detected. The first two columns of the Excel spreadsheet contain the AI ​​texturing image input by the user and the prompt input when the texturing image model generates the image. The last two columns of the Excel spreadsheet contain the content extracted and populated into the Excel spreadsheet based on the detection results input from the FastAPI service module.

[0165] Table 1

[0166]

[0167]

[0168] Taking the application scenario of bad case mining in on-site project algorithms as an example, in some implementations, users fill in an Excel spreadsheet with the images of the scene to be detected with alarm boxes, the basic prompts input by other large models when identifying alarm content in the images, and related event definitions through a client. The FastAPI service module reads the Excel spreadsheet, generates the corresponding prompt based on the read content and the pre-set prompt template, and calls the multimodal large model. The multimodal large model evaluates the images of the scene with alarm boxes based on the prompt.

[0169] For example, the user inputs a basic prompt: "Based on the national standard definition of urban management incidents, issue an alert for exposed garbage in the image and mark it with a red box." The user also inputs alert images from other large models annotated based on this basic prompt. The alert image is as follows: Figure 4 As shown. The detection system proposed in this embodiment detects whether the alarm image is correct, that is, whether the alarm is a bad case. For example, if the detection result generated by the detection system proposed in this embodiment is "exposed garbage exists within the red box in the image", then the alarm image is correct, and the alarm does not belong to a bad case.

[0170] In some implementations, the multimodal large model proposed in this disclosure supports multi-turn sessions and can achieve strategy optimization through parameter pass-through, further improving the recognition accuracy of the multimodal large model for complex scene images.

[0171] Taking image annotation evaluation as an example, a multimodal large model is used to evaluate the annotations of the image annotation team. For instance, the user organizes the images to be evaluated into an Excel spreadsheet. The FastAPI service module reads this Excel spreadsheet, extracts the images to be evaluated, and generates a multi-turn dialogue prompt based on a pre-defined prompt template. Below is an example of a multi-turn prompt design and the data returned by the multimodal large model, including:

[0172] First prompt: You need to determine if anyone in the picture is using a mobile phone or making a call. Are there people in the picture? If so, what are they doing?

[0173] The model's first round of data return: In this image, I see one person using a mobile phone. He is standing inside the store, the phone in his hand, and it appears he is making a call. There are also other people inside the store, but they are not using mobile phones or making calls. These people may be browsing merchandise, shopping, or interacting with store clerks.

[0174] Second prompt: Are these people doing things that might involve using a mobile phone?

[0175] The model's second round of data returned: These people may be browsing products, shopping, or interacting with store clerks. They may be using their phones for these activities, but in this image, only that one person is actually using a phone.

[0176] The third prompt: Your answer must be in Chinese and output in JSON code format, for example: {'phone':1,'reason':'Someone is using a mobile phone or making a call in the picture'} (0 means no behavior was identified; 1 means the behavior of using a mobile phone or making a call was identified).

[0177] The model returned data in the third round:

[0178] #1 point, Reason: The image shows someone using a mobile phone or making a call; the re-judgment result is accurate.

[0179] json\n{\n"phone\":1,\n"reason\":\"There is someone using a mobile phone or making a call in the picture\n}

[0180] After the multimodal large model returns the above detection results, the FastAPI service module organizes the detection results, for example, by organizing them into text information to describe standard information, and fills the text information into an Excel spreadsheet, and then returns the Excel spreadsheet with the text information filled in to the client.

[0181] In summary, the detection method provided in this disclosure integrates the capabilities of a multimodal large model, enabling functions such as quality assessment of AI text-generated image data, identification of algorithmic bad cases in project scenarios, and evaluation of images labeled by image annotation teams. Integrating a multimodal large model on the engineering side achieves a fully automated image quality assessment process, reducing manual intervention and improving work efficiency and assessment quality. Automated quality assessment using a multimodal large model significantly reduces the workload and time required for manual review, improving efficiency and reducing costs. Furthermore, the use of a multimodal large model, through machine learning and self-training, reduces assessment bias caused by human factors, improving the objectivity and accuracy of the assessment. The multimodal large model can process large amounts of image data, is not limited by the amount of data required for manual review, and can handle large-scale image quality assessment tasks. Moreover, the multimodal large model can continuously improve the accuracy of image quality assessment through continuous learning and optimization, adapting to constantly changing image quality standards.

[0182] This disclosure also proposes a detection system based on a large model. Figure 5 This is a schematic diagram of the structure of a large-model-based detection system 500 according to an embodiment of the present disclosure, including a web application framework and a multimodal large model, wherein,

[0183] The Web application framework 510 is used to receive a detection instruction from a client, which contains the content to be detected; determine the detection type based on the detection instruction, and call the multimodal large model based on the detection type; and also to format the detection results returned by the multimodal large model and return the formatted detection results to the client.

[0184] The multimodal large model 520 is used to detect the content to be detected based on the call and return the detection results to the web application framework.

[0185] In some implementations, the detection types include at least one of AI-generated image data quality assessment, algorithm scene error detection, and image annotation assessment.

[0186] In some implementations, the multimodal large model 520 includes at least two application interfaces, each corresponding to a detection capability of a detection type.

[0187] In some implementations, the Web application framework 510 is used to call application interfaces based on detection type to enable the invocation of multimodal large models;

[0188] Among them, the application interface called by the Web application framework 510 corresponds to the detection capability of the detection type.

[0189] In some implementations, the detection instruction includes a table of a predetermined format, which contains multiple items to be detected.

[0190] In some implementations, the Web application framework 510 is used for:

[0191] Extract tables with a predefined format;

[0192] According to a predetermined strategy, extract the content of a specified column from a table with a predetermined format, the content of which includes images and / or text;

[0193] The extracted images and / or text content are stored in the local service path, and the format of the extracted images and / or text content is converted.

[0194] In some implementations, the Web application framework 510 is used for:

[0195] Based on the detection type, retrieve the pre-saved prompt word template corresponding to that detection type;

[0196] Based on the prompt word template and extracted content, prompt words are generated to instruct the multimodal large model 520 to perform detection;

[0197] Invoke the multimodal large model 520 and send the generated prompt words to the multimodal large model 520.

[0198] In some implementations, the multimodal large model 520 is used for:

[0199] Based on the received prompts, each piece of content to be detected is detected separately;

[0200] Return the detection results for each piece of content to be detected to the Web application framework, according to at least one of the output formats required by the prompt words and the output examples.

[0201] In some implementations, the Web application framework 510 is used for:

[0202] The detection results of each piece of content to be detected are segmented into text to obtain at least one of the detection score and the scoring reason;

[0203] Write at least one of the detection score and scoring reason for each item to be detected into the table in the predetermined format to obtain the table after filling in the detection results;

[0204] Return the form containing the test results to the client.

[0205] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0206] The acquisition, storage, and application of personal information by users involved in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0207] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0208] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0209] like Figure 6As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0210] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0211] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as detection methods. For example, in some embodiments, the detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the detection method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the detection method by any other suitable means (e.g., by means of firmware).

[0212] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0213] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0214] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0215] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0216] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0217] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0218] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0219] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A detection method based on a large model, comprising: The web application framework receives a detection instruction, which contains a table in a pre-formatted format, and the table in the pre-formatted format contains multiple items to be detected; The web application framework determines the detection type based on the content to be detected in the detection indication, and calls a multimodal large model based on the detection type; the detection type includes AI text-based image data quality assessment, algorithm scene error mining, and image annotation assessment; wherein, the AI ​​text-based image data quality assessment includes quality assessment of images generated by the text-based image model, the algorithm scene error mining includes mining error cases in alarm data detected by other large models from images in the on-site project algorithm scene, and the image annotation assessment includes evaluation of the data annotated by the image annotation team for the images; The multimodal large model detects the content to be detected based on the call; The web application framework receives the detection results returned by the multimodal large model; The Web application framework performs text segmentation on the detection results of each of the items to be detected to obtain at least one of the detection score and the scoring reason; and writes at least one of the detection score and the scoring reason for each of the items to be detected into the table in the predetermined format to obtain the table after filling in the detection results. Send the form after filling in the test results.

2. The method according to claim 1, further comprising, before receiving the detection indication: Pre-train and deploy a large multimodal model that provides detection capabilities for at least two of the aforementioned detection types; Establish at least two application interfaces, each of which corresponds to a detection capability of the detection type.

3. The method according to claim 2, wherein, The step of calling the multimodal large model based on the detection type includes: The application interface is called based on the detection type to enable the calling of the multimodal large model; The application interface called corresponds to the detection capability of the detection type.

4. The method according to claim 1, wherein, Before determining the detection type based on the content to be detected in the detection indication, the method further includes: Extract the table with the predetermined format; According to a predetermined strategy, the content of a specified column is extracted from a table of a predetermined format, wherein the content of the specified column includes images and / or text; The extracted images and / or text content are stored in a local service path, and the extracted images and / or text content are converted to the desired format.

5. The method according to claim 4, wherein, The step of calling the multimodal large model based on the detection type includes: Based on the detection type, obtain the pre-saved prompt word template corresponding to the detection type; Based on the prompt word template and the extracted content, prompt words are generated to instruct the multimodal large model to perform detection; The multimodal large model is invoked, and the generated prompt words are sent to the multimodal large model.

6. The method according to claim 5, wherein, The multimodal large model detects the content to be detected based on the call, including: The multimodal large model detects each of the contents to be detected based on the received prompt words; The multimodal large model determines the detection result of each of the contents to be detected according to at least one of the output format required by the prompt words and the output examples.

7. A detection system based on a large model, comprising a web application framework and a multimodal large model, wherein, The web application framework is used to: receive a detection instruction from a client, the detection instruction containing a table in a predetermined format, the table containing multiple items to be detected; determine a detection type based on the items to be detected in the detection instruction, and invoke the multimodal large model based on the detection type; the detection type includes AI text-based image data quality assessment, algorithm scene error mining, and image annotation evaluation; wherein, the AI ​​text-based image data quality assessment includes quality assessment of images generated by the text-based image model, the algorithm scene error mining includes mining error cases in alarm data detected by other large models from images in on-site project algorithm scenarios, and the image annotation evaluation includes evaluating the data annotated by the image annotation team for the images; further, it is used to perform text segmentation on the detection results returned by the multimodal large model to obtain at least one of a detection score and a scoring reason; write at least one of the detection scores and scoring reasons for each item to be detected into the table in the predetermined format to obtain a table with filled detection results; and return the table with filled detection results to the client; The multimodal large model is used to detect the content to be detected based on the call, and return the detection results to the Web application framework.

8. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images