Crop fruit quality detection method and system based on multi-modal large model

Abstract

The invention discloses a crop fruit quality detection method and system based on a multi-modal large model. The method comprises the following steps: establishing the multi-modal large model for crop fruit quality detection based on an LISA structure; the method comprises the following steps: preprocessing an input crop fruit image by using an image preprocessing agent, and then inputting the preprocessed crop fruit image into a semantic segmentation agent to extract a fruit region; inputting the crop fruit image subjected to semantic segmentation into a semantic segmentation review agent, reviewing semantic segmentation by comparing the features of the actual crop fruit with the extracted fruit region, if the review result is that the fruit is the fruit, inputting the fruit into a quality detection agent, otherwise, carrying out semantic segmentation again; and the quality detection agent detects the quality of the fruit area and outputs a detection result and natural language description of a detection basis. According to the method, the accuracy and efficiency of fruit maturity evaluation can be improved, meanwhile, clear natural language description is generated, and the understandability of an output result is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of multimodal large models, and in particular to a method and system for detecting the quality of crop fruits based on the multimodal large model. Background Art

[0002] The quality assessment of crop fruits plays a key role in choosing the right time to pick, reducing unnecessary waste and ensuring market supply. The traditional manual picking detection method requires a lot of manpower and material resources and is inefficient. With the development of artificial intelligence, more and more methods based on machine learning, deep learning, and target detection are being applied to the quality detection of crop fruits. Although the detection efficiency has been improved to a certain extent, the following problems still exist:

[0003] (1) The accuracy of the results is heavily dependent on the quantity and quality of training data: In machine learning, deep learning, and target detection methods, the amount of training data directly affects the generalization ability and accuracy of the model. Insufficient training data is very likely to lead to underfitting, that is, the model cannot effectively learn the features of different maturity stages, thus affecting the accuracy of judgment. The quality of training data is also crucial. Data quality includes image clarity, annotation accuracy, and sample diversity. Low-quality data (such as blurred images, mislabeled samples, class imbalanced samples, or unrepresentative images) will cause the model to learn noisy or biased features, thereby affecting its judgment ability. Sufficient and high-quality labeled data requires a lot of manual collection and annotation work, which will undoubtedly introduce a lot of manpower and material resources.

[0004] (2) The judgment results lack sufficient interpretability: In deep learning and target detection methods, the model structure is usually a black box model due to its complex internal feature extraction process, and the output results lack intuitive explanation. Although some algorithms such as random forests, gradient boosting decision trees and their subsequent improved algorithms have certain intrinsic interpretability, these interpretabilities are all at the feature level and are not clear and intuitive enough. Summary of the invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a method and system for detecting the quality of crop fruits based on a multimodal large model. The multimodal large model is fine-tuned through instructions of a small number of samples to efficiently detect the quality of crop fruits in pictures, including maturity detection, disease and pest detection, etc., and output a natural language explanation of the detection results, thereby solving the problems of heavy reliance on training data and lack of interpretability of judgment results in the prior art.

[0006] Technical solution: The method for detecting the quality of crop fruits based on a multimodal large model described in the present invention establishes a multimodal large model for detecting the quality of crop fruits based on a LISA structure; wherein, an image preprocessing agent is used to preprocess the input crop fruit image; the preprocessed crop fruit image is input into a semantic segmentation agent to extract the fruit area through semantic segmentation; the semantically segmented crop fruit image is input into a semantic segmentation review agent, and the semantic segmentation is reviewed by comparing the features of the actual crop fruit with the extracted fruit area. If the review result is that it is a fruit, the semantically segmented crop fruit image is input into a quality detection agent, otherwise it is returned to the semantic segmentation agent to re-semantically segment; the quality detection agent detects the quality of the fruit area, and outputs the detection result and a natural language description of the detection basis, wherein the quality includes maturity and pests and diseases.

[0007] Furthermore, the multimodal large model for crop fruit quality detection based on the LISA structure includes: collecting a number of crop fruit pictures and performing semantic segmentation and annotation, establishing a data set, constructing an instruction fine-tuning data set based on the data set, and performing instruction fine-tuning on the multimodal large model.

[0008] Furthermore, in the image preprocessing agent, preprocessing prompt words are set to detect whether the input crop fruit image has noise and / or unclear edges and / or non-prominent entities. If so, corresponding noise elimination and / or edge sharpening and / or contrast enhancement preprocessing operations are performed, and the preprocessed crop fruit image is input into the semantic segmentation agent; otherwise, the crop fruit image is directly input into the semantic segmentation agent.

[0009] Furthermore, in the semantic segmentation agent, initial prompt words are set to perform semantic segmentation on the preprocessed crop fruit image to extract the fruit area; review prompt words are set to perform semantic segmentation again on the preprocessed crop fruit image in combination with the review results of the semantic segmentation review agent to extract the fruit area.

[0010] Furthermore, in the semantic segmentation review agent, review prompt words are set, and the semantic segmentation is reviewed by comparing the characteristics of actual crop fruits with the extracted fruit area, and the review results are output. The review results include the judgment result and judgment basis of whether the fruit area is a fruit entity, as well as the natural language evaluation of the fruit area.

[0011] Furthermore, the method also includes a finite state machine, which manages the calling relationship and execution order between the agents according to the preset state definitions and state transition conditions. The finite state machine adopts the state transition matrix method to efficiently manage and control the transition between different states, thereby effectively avoiding circular calls or invalid calls.

[0012] The state definitions include:

[0013] Initial state: In the system preparation stage, all agents are in an inactive state;

[0014] Image preprocessing state: the image preprocessing agent is activated to perform image optimization tasks;

[0015] Semantic segmentation state: After the image preprocessing is completed, it is transferred to the image semantic segmentation agent to perform the task;

[0016] Review status: After the semantic segmentation is completed, it enters the blueberry fruit image semantic segmentation review agent for verification;

[0017] Maturity judgment status: After passing the review, the blueberry fruit maturity judgment agent is activated to make the final judgment;

[0018] End state: All tasks are completed and the system outputs the final results.

[0019] The state transition conditions include:

[0020] From the initial state to the image preprocessing state: when the system receives the raw image input, it automatically transfers;

[0021] From the image preprocessing state to the semantic segmentation state: When the image preprocessing agent outputs the optimized image, it moves to the next state;

[0022] From the semantic segmentation state to the review state: After the semantic segmentation agent completes the task, the output result triggers the activation of the review agent;

[0023] From the review state to the semantic segmentation state: If the review agent finds that the segmentation result is insufficient, it returns to the semantic segmentation state for improvement;

[0024] From the review state to the maturity judgment state: when the review result is satisfactory, it is transferred to the maturity judgment agent;

[0025] From maturity judgment state to end state: When the maturity judgment is completed and the results are output, the system enters the end state.

[0026] Furthermore, the method also includes setting a global maximum number of retries, and each time the semantic segmentation agent or the semantic segmentation review agent is executed, a counter is incremented by one, and when the counter reaches the global maximum number of retries, the semantically segmented crop fruit image in the current semantic segmentation agent is output.

[0027] The global maximum number of retries includes the following points:

[0028] Initialize retry counter: When the process starts, a global retry counter is initialized and incremented every time a call or re-call is made between the image semantic segmentation agent and the blueberry fruit image semantic segmentation review agent.

[0029] Set the maximum retry count threshold: Set a reasonable maximum retry count threshold (for example, 5 times). When the retry counter reaches or exceeds this threshold, the system stops the image semantic segmentation and review loop and enters the exception handling process.

[0030] Exception handling process: If the number of retries reaches the maximum value, the exception handling mechanism will be triggered. The exception situation can be recorded for subsequent debugging and analysis, and the current best segmentation result will be output. The remaining segmentation and review steps will be skipped to prevent the system from falling into an infinite loop.

[0031] By setting the global maximum number of retries, the problem of circular calls in the semantic segmentation process of blueberry fruit images can be effectively prevented, ensuring the efficiency and stability of the entire system.

[0032] The crop fruit quality detection system based on a multimodal large model of the present invention comprises a multimodal large model established based on a LISA structure, and is used for detecting the quality of crop fruits; wherein, an image preprocessing agent is used to preprocess an input crop fruit image; the preprocessed crop fruit image is input into a semantic segmentation agent to extract a fruit region through semantic segmentation; the semantically segmented crop fruit image is input into a semantic segmentation review agent, and the semantic segmentation is reviewed by comparing the features of actual crop fruits with the extracted fruit region; if the review result is that the crop fruit image is a fruit, the semantically segmented crop fruit image is input into a quality detection agent, otherwise, the semantic segmentation is returned to the semantic segmentation agent for re-semantic segmentation; the quality detection agent detects the quality of the fruit region, and outputs a natural language description of the detection result and the detection basis, wherein the quality includes maturity and pests and diseases.

[0033] The electronic device of the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the method for detecting the quality of crop fruits based on a multimodal large model is implemented.

[0034] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the method for detecting the quality of crop fruits based on a multimodal large model is implemented.

[0035] Beneficial effects: Compared with the prior art, the advantages of the present invention are:

[0036] (1) Compared with the traditional method that relies on a large amount of manual collection and annotation, the present invention can improve the generalization ability of the model through the synergy of intelligent agents on the basis of relatively less high-quality data, thereby reducing the consumption of manpower and material resources. Through the division of labor and cooperation of intelligent agent modules, combined with image preprocessing, semantic segmentation and review, the output quality of each stage can be gradually improved. This multi-stage review mechanism ensures the accuracy of the final result and reduces the underfitting problem caused by insufficient or poor quality training data in traditional methods.

[0037] (2) The present invention introduces a decision-making process based on prompt words, and the intelligent agent generates a clear natural language description when making a judgment. This description not only enhances the comprehensibility of the output results, but also provides an intuitive basis for users, making up for the shortcomings of the black box characteristics of the deep learning model, so that users can understand the basis for judgment. The present invention can be applied to multiple fields such as agricultural modernization, intelligent agricultural management and food safety testing, and has broad application prospects. For example, in the field of agricultural modernization, this method can monitor the maturity of fruits in real time through image processing technology, and combine the collaborative working characteristics of multiple intelligent agents to achieve efficient analysis and accurate judgment of the fruit state. Using sensor nodes and image acquisition equipment, the maturity state of fruits can be evaluated in time to ensure that agricultural practitioners can pick them at the best time, thereby improving the quality and yield of fruits. This method enhances the scientificity and reliability of maturity judgment through the application of multimodal large models, and has significant promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A multi-agent module structure diagram of an embodiment of the present invention;

[0039] Figure 2 This is a flowchart of fine-tuning a multi-modal large model according to an embodiment of the present invention;

[0040] Figure 3 A schematic diagram of a flow chart of an image preprocessing subtask according to an embodiment of the present invention;

[0041] Figure 4 A schematic diagram of the process of the blueberry entity semantic segmentation subtask according to an embodiment of the present invention;

[0042] Figure 5 A flowchart of a blueberry entity semantic segmentation review subtask according to an embodiment of the present invention;

[0043] Figure 6 This is a flow chart of a blueberry entity maturity judgment subtask according to an embodiment of the present invention;

[0044] Figure 7 Schematic diagram of a state transition matrix of a finite state machine according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0046] The method for detecting the quality of crop fruits based on a multimodal large model of the present invention comprises a multimodal large model for fine-tuning instructions of a small number of samples, a multi-agent module, and a finite state machine. The multimodal large model as a base contains a wide range of knowledge, provides the ability to understand and reason about image and text category input, and the effect of fine-tuning instructions through a small number of samples is better; Figure 1 As shown in the figure, the multi-agent module consists of four agents, each of which works together to perform specific subtasks in the task of crop fruit quality detection. The agents use the above capabilities of the multimodal large model to make decisions and perform specified tasks according to the output of the multimodal large model in different tasks; the finite state machine is used to manage the calling relationship and execution order between the agents. It ensures the sequence and consistency of the entire process, prevents circular calls or invalid calls between agents, provides a clear decision path, and ensures that the dependencies of the agents in the task of crop fruit quality detection are strictly enforced.

[0047] This embodiment takes the detection of blueberry fruit maturity as an example for detailed description.

[0048] 1. Multimodal Large Model

[0049] The instructions for fine-tuning multimodal large models with a small number of samples partially utilize the existing open source LISA structure, such as Figure 2 As shown, the fine-tuning part includes the following steps:

[0050] Step 101, collect 30 blueberry fruit pictures, preferably including multiple blueberry entities and different categories (ripe, semi-ripe, unripe), and the number of categories is roughly equal to ensure the quantity balance of category samples.

[0051] Step 102: Use the labelImg annotation tool to perform semantic segmentation annotation on the image collected in step 101.

[0052] Step 103, construct instruction fine-tuning data, the template is:

[0053] {"Question":"Please identify all the ripe blueberries, half-ripe blueberries and unripe blueberries in the picture","Answer":"[ID1] is a ripe blueberry, [ID2] is also a ripe blueberry, [ID2] is a half-ripe blueberry, and no unripe blueberries were found"}, or {"Question":"Please identify all the ripe blueberries, half-ripe blueberries and unripe blueberries in the picture","Answer":"[ID1] is a half-ripe blueberry, [ID2] is an unripe blueberry"};

[0054] Among them, ID1, ID2, and ID3 are the IDs of the blueberry categories in the picture, specifically, the annotation masks of the blueberry entities of each category in the picture. Question is the field representing the input question in the instruction fine-tuning data, and Answer is the field representing the result that should be output in the instruction fine-tuning data.

[0055] Step 104, using the instruction fine-tuning dataset constructed in step 103, use LORA technology to fine-tune the multimodal large model to inject and enhance its ability to semantically segment blueberry fruits and judge maturity.

[0056] Step 105, complete the instruction fine-tuning and use it as part of the multi-modal large model of this embodiment.

[0057] The multimodal large model serves as the brain of subsequent intelligent agents, giving them the ability to handle subtasks of judging the maturity of blueberries at different stages. Instruction fine-tuning is not necessary. Experiments have shown that the semantic segmentation capability and classification accuracy of blueberry entities can be significantly improved with only a small amount of instruction fine-tuning.

[0058] 2. Multi-agent Module

[0059] The multi-agent overall module sets up four types of agents: image preprocessing agent, semantic segmentation agent, semantic segmentation review agent, and quality inspection agent. The input of the multi-agent overall module is a picture that may contain blueberry entities input by the user. In the first part, the picture is first preprocessed by the image preprocessing agent to enhance the recognizable features in the image; the second part is the semantic segmentation agent, which uses the output of the image preprocessing agent as input to perform semantic segmentation on it to detect and extract the blueberry fruit entity in the image; the third part is the semantic segmentation review agent, which uses the output of the semantic segmentation agent as input to review whether the candidate blueberry entity extracted by the semantic segmentation agent is correct. If it thinks it is correct, it returns correct, otherwise it outputs the review result and returns the result to the semantic segmentation agent, allowing it to re-extract based on the review evaluation; the fourth part is the quality inspection agent, which uses the final output of the semantic segmentation agent as input to perform a multi-classification task with three categories, and the category labels are: ripe, half-ripe, and unripe, and outputs the natural language expression of its judgment basis to explain the judgment basis.

[0060] like Figure 3The figure shows a flow chart of the image preprocessing agent of the present embodiment performing the image preprocessing subtask, whose input is the original blueberry fruit image, which may have various problems, such as noise, blur, poor contrast or inappropriate resolution. The agent receives these unprocessed original images and makes corresponding processing decisions based on the image characteristics. The agent obtains the steps that the multimodal large model needs to perform preprocessing on the image through the prompt words, and calls predefined tools to perform processing. The tools include a noise removal tool to remove background noise in the image and improve clarity. An edge sharpening tool is used to enhance the boundaries of blueberry fruits in the image to make the boundaries clearer. A contrast enhancement tool is used to optimize the contrast of the image to make the fruit features more prominent. The image preprocessing subtask flow is described as follows:

[0061] Step 201, set the prompt words of the image preprocessing agent, the preprocessing prompt words are:

[0062] [{'role':'system','content':'You are an experienced image processing expert specializing in related fields. You should judge Is there noise, unclear edges, and weak entities in the image? If so, call the following tools, the tool list is: clarity tool, edge sharpening tool, feature highlighting tool'},{'role':'user','content':'For the input blueberry fruit original image, first perform noise removal to remove background interference and improve image clarity. Next, apply edge sharpening technology to enhance the boundary features of the fruit and ensure that the outline is clear. Finally, adjust the contrast and brightness of the image to make the color of the blueberry fruit more vivid and the details more prominent, and output a high-quality image suitable for subsequent semantic segmentation processing. '}].

[0063] The role field indicates the role corresponding to the prompt word, and the value of the role field can be system indicating a system role or user indicating a user role. The content indicates the description of the specific role and task. The labels are represented as input image data.

[0064] Step 202, define and set an image noise removal code as a definition enhancement tool, the output of which is a noise-removed image.

[0065] Step 203, define and set the image edge sharpening code as an edge sharpening tool, the output of the tool is an edge-sharpened image.

[0066] Step 204, define and set an image contrast enhancement code as a feature highlighting tool, the output of which is an image with enhanced image contrast.

[0067] Step 205, the image preprocessing agent determines whether the input image is clear enough. If it is clear, it proceeds to step 206, otherwise it proceeds to step 202.

[0068] Step 206 , the image preprocessing agent determines whether the image edge needs to be sharpened based on the input image. If it is determined that sharpening is required, proceed to step 203 , otherwise proceed to step 207 .

[0069] Step 207 , judging whether the contrast needs to be adjusted according to the input image, if yes, proceed to step 304 , otherwise go to step 208 .

[0070] Step 208: The process ends and the image is output.

[0071] like Figure 4 The figure shows a flow chart of the semantic segmentation agent performing the blueberry entity semantic segmentation subtask. The input is the image preprocessed by the image preprocessing agent. The agent uses a multimodal large model, and the preset prompt words are divided into two parts to deal with two different scenarios. The prompt words are: initial prompt words, which are used for the scene output by the image preprocessing agent in the previous step; review prompt words, which are used for the scene output by the semantic segmentation review agent in the previous step. The prompt words are used to understand the information about the object blueberry in the input, infer and identify the possible positions and contours of blueberries in the image; the output is a semantic segmentation image of the blueberry fruit, in which the fruit area in the image has been clearly marked and has accurate contour information. The flow of the blueberry entity semantic segmentation subtask is described as follows:

[0072] Step 301, set the initial prompt word of the semantic segmentation agent, the initial prompt word is:

[0073] [{'role':'system','content':'You are an experienced agricultural and forestry expert specializing in fruit identification. You should determine whether there are blueberries in the image and mark their location and outline. Please ensure that the fruit boundaries are clear and the details are accurate. '},{'role':'user','content':'For the input blueberry fruit image, perform semantic segmentation tasks, identify all blueberry areas, and mark their accurate boundaries to ensure that the fruit areas are clear. '}].

[0074] Step 302, setting the review prompt words of the semantic segmentation agent, the review prompt words are:

[0075] [{'role':'system','content':'You are an experienced agricultural and forestry expert specializing in fruit identification. You should determine whether there are blueberries in the image and mark their location and outline. Please ensure that the fruit boundaries are clear and the details are accurate. '},{'role':'user','content':'###RequirementsYou need to re-judge the segmentation of the blueberry area in the image to improve the accuracy based on the review information provided in the AuditResults section. The same requirements are: perform semantic segmentation tasks, identify all blueberry areas, mark their accurate boundaries, and ensure that the fruit areas are clear. ###Audit Results{audit_result_placeholder})]. '}].

[0076] This prompt is written in a segmented manner. ###Requirements represents the requirement paragraph, which is used to clarify the specific tasks. ###Audit Results represents the audit results paragraph, where {audit_result_placeholder} is a placeholder that will be filled with specific audit result data during the process.

[0077] In step 303, the first input received is the output from the image preprocessing agent. Therefore, if the input only contains an image, the process should proceed to step 304. Otherwise, if the input contains both an image and review information, the input is the output from the semantic segmentation review agent. This means that the semantic segmentation review agent is not satisfied with the result of the semantic segmentation and needs to be re-evaluated and segmented in combination with the review information. Therefore, the process should proceed to step 305.

[0078] Step 304, after receiving the input image, the semantic segmentation agent uses the initial prompt words set in step 301 in combination with the multimodal large model to identify the fruit area of ​​the image, extracts a list of candidate blueberry entities, outputs the semantically segmented image, and goes to step 306.

[0079] Step 305: After the semantic segmentation agent receives the image input and the review result input, it first replaces the {audit_result_placeholder} placeholder part in the review prompt word set in step 302 with the result of the review result input. The review result includes the evaluation statement of the semantic segmentation review agent on the dissatisfaction with the previous semantic segmentation result. Then, the fruit area is identified again, the semantically segmented image is output, and then go to step 306.

[0080] Step 306, the extracted candidate blueberry entity list is passed to the next agent, and the process ends.

[0081] like Figure 5 The figure shows the process flow of the semantic segmentation review agent performing the blueberry entity semantic segmentation review subtask. The input of the agent is the semantic segmentation image completed by the semantic segmentation agent. The multimodal large model is used to perform preliminary verification of the segmentation results to ensure that the final blueberry entity recognition results are highly reliable and accurate. The process flow of the blueberry entity semantic segmentation review subtask is described as follows:

[0082] Step 401, set the prompt words of the semantic segmentation review agent, and the review prompt words are:

[0083] [{'role':'system','content':'You are an experienced agricultural and forestry expert who specializes in the field of fruit judgment. You are required to provide accurate, neutral and unbiased answers. '},{'role':'user','content':'###RequirementsYou need to determine whether each image in the image list in the Segmentations section is a blueberry, and use human-understandable text as the basis for judgment. The answer must be in Python List format. The number of elements in the List is consistent with the number of images to be judged in the Segmentations section. Each element is the judgment result of the corresponding subscript image. Here is an example [(True, the fruit in the image segmentation is dark purple in color, and there is a layer of fine white frost on the surface of the peel, which makes the blueberry look a bit matte. There is a unique small "crown" calyx on the top of the fruit), (False, the fruit in the image segmentation does not meet the color and characteristics that blueberries should present. ###Segmentations{segmentations_placeholder})]'}].

[0084] The ###Segmentations paragraph represents the specific segmentation paragraph, and {segmentations_placeholder} is the placeholder for the segmentation data, which will be filled with the specific segmentation data in the process.

[0085] Step 402, by comparing the actual blueberry features with the segmentation results of the input semantic segmentation result image, possible errors and omissions are identified. A detailed review report is generated, pointing out the advantages and disadvantages of the segmentation results, and making suggestions for improvement. Its output is a list, in which the number of elements is consistent with the number in the candidate blueberry entity list output by the image semantic segmentation agent. Each element in the judgment result list represents the judgment result for the candidate blueberry fruit entity. Each element is a python Tuple type, consisting of two elements. The first element is a Bool value type variable, indicating whether the blueberry fruit image semantic segmentation review agent judges the candidate entity as a blueberry. The second element is a string type variable, describing the specific judgment basis and advantages and disadvantages as well as a detailed review report.

[0086] Step 403, the process ends and returns to the judgment result list output in step 402.

[0087] like Figure 6 The figure shows a flow chart of the quality inspection agent executing the blueberry entity maturity judgment subtask. The input is the output of the semantic segmentation agent after the last review and improvement by the semantic segmentation review agent. The agent uses a multimodal large model and combines the ideas of Few shots and thinking chains to better stimulate the multimodal large model's reasoning ability for this task. It forces the output of three categories of multi-classification tasks, with category labels: ripe, half-ripe, and unripe, and outputs a natural language expression of the judgment basis to explain the judgment basis. In practical applications, if pest and disease detection is performed, the output category label can be a binary classification label, with the labels being: with pests and without pests. The flow of the blueberry fruit maturity judgment subtask is described as follows:

[0088] Step 501, set the prompt words of the quality detection agent, the detection prompt words are:

[0089] [{'role':'system','content':'You are a professional fruit maturity assessment expert, specializing in the judgment of blueberry fruit. You need to input image information For maturity classification, you are required to provide accurate, neutral, and unbiased answers. '},{'role':'user','content':'###RequirementsPlease classify the maturity of blueberries in the image according to the image in the Segmentations section. The values ​​of the maturity category label can only be: ripe, half-ripe, and unripe. The result outputs a Python tuple, which contains only two elements. The first element corresponds to a judgment result, and the second element is a natural language description of the judgment basis. For example: (ripe, blueberry fruit color is dark purple, the surface is smooth, and the shape is full), (half-ripe, blueberry fruit color is green, and the edges are a little yellow), (unripe, blueberry fruit color is green or light green, lacking obvious blue or purple.). ###Segmentations{segmentations_placeholder}'}].

[0090] Step 502, accepting the list outputted by the semantic segmentation agent for the last time as input, looping through all elements in the list (including the semantic segmentation images of blueberry entities), if the iteration is not completed, proceeding to step 503, if the iteration of all elements is completed, proceeding to step 504;

[0091] Step 503, combining the detection prompt words set in step 501 to judge the segmented area of ​​the current iteration element, and output it in Python tuple format to represent the judgment result. The template is as follows:

[0092] (Classification results, natural language description of the basis for judgment);

[0093] The value of the first element of the tuple is one of ripe, half-ripe, and unripe, indicating the maturity judgment result of the blueberry in this embodiment. The second element of the tuple is a natural language description of the judgment basis.

[0094] Step 504, the process ends, and the result of each iteration of step 503 is used as the final result of this embodiment.

[0095] (III) Finite State Machine

[0096] The finite state machine is used to manage the calling relationship and execution order between the agents, ensuring the order and consistency of the entire detection process. The state machine controls the activation and execution process of the agent by defining each state and its transition conditions, thereby effectively avoiding circular calls or invalid calls. The state machine is implemented using the state transition matrix method to efficiently manage and control the transition between different states. Figure 7 Shown is the state transfer matrix of the finite state machine.

[0097] The crop fruit quality detection system based on a multimodal large model of the present invention comprises a multimodal large model established based on a LISA structure, and is used for detecting the quality of crop fruits; wherein, an image preprocessing agent is used to preprocess an input crop fruit image; the preprocessed crop fruit image is input into a semantic segmentation agent to extract a fruit region through semantic segmentation; the semantically segmented crop fruit image is input into a semantic segmentation review agent, and the semantic segmentation is reviewed by comparing the features of actual crop fruits with the extracted fruit region; if the review result is that the crop fruit image is a fruit, the semantically segmented crop fruit image is input into a quality detection agent, otherwise, the semantic segmentation is returned to the semantic segmentation agent for re-semantic segmentation; the quality detection agent detects the quality of the fruit region, and outputs a natural language description of the detection result and the detection basis, wherein the quality includes maturity and pests and diseases.

[0098] The electronic device of the present invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the computer program is loaded into the processor, the method for detecting the quality of crop fruits based on a multimodal large model is implemented.

[0099] The computer-readable storage medium of the present invention stores a computer program, and when the computer program is executed by a processor, the method for detecting the quality of crop fruits based on a multimodal large model is implemented.

[0100] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store program code in the form of instructions or data structures and that can be accessed by a computer.

[0101] The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.

Claims

1. A method for detecting crop fruit quality based on a multimodal large model, characterized in that: A multimodal large model for crop fruit quality detection is established based on the LISA structure; wherein, an image preprocessing agent is used to preprocess the input crop fruit image; the preprocessed crop fruit image is input into a semantic segmentation agent to extract the fruit area through semantic segmentation; the semantically segmented crop fruit image is input into a semantic segmentation review agent, and the semantic segmentation is reviewed by comparing the features of the actual crop fruit with the extracted fruit area. If the review result is that it is a fruit, the semantically segmented crop fruit image is input into a quality detection agent, otherwise it is returned to the semantic segmentation agent for re-semantic segmentation; the quality detection agent detects the quality of the fruit area, and outputs the detection result and a natural language description of the detection basis, wherein the quality includes maturity and pests and diseases.

2. The method for detecting crop fruit quality based on a multimodal large model according to claim 1, characterized in that: The multimodal large model for crop fruit quality detection based on the LISA structure includes: Several pictures of crop fruits are collected and semantically segmented and annotated to establish a data set. An instruction fine-tuning data set is constructed based on the data set to perform instruction fine-tuning on the multimodal large model.

3. The method for detecting crop fruit quality based on a multimodal large model according to claim 1, characterized in that: In the image preprocessing agent, preprocessing prompt words are set to detect whether the input crop fruit image has noise and / or unclear edges and / or non-prominent entities. If so, corresponding noise elimination and / or edge sharpening and / or contrast enhancement preprocessing operations are performed, and the preprocessed crop fruit image is input into the semantic segmentation agent. Otherwise, the crop fruit image is directly input into the semantic segmentation agent.

4. The method for detecting crop fruit quality based on a multimodal large model according to claim 1, characterized in that: In the semantic segmentation agent, an initial prompt word is set to perform semantic segmentation on the preprocessed crop fruit image to extract the fruit area; A review prompt word is set, and semantic segmentation is performed again on the preprocessed crop fruit image in combination with the review result of the semantic segmentation review intelligent agent to extract the fruit area.

5. The method for detecting crop fruit quality based on a multimodal large model according to claim 1, characterized in that: In the semantic segmentation review agent, review prompt words are set, and the semantic segmentation is reviewed by comparing the characteristics of actual crop fruits with the extracted fruit area, and the review results are output. The review results include the judgment result and judgment basis of whether the fruit area is a fruit entity, as well as the natural language evaluation of the fruit area.

6. The method for detecting crop fruit quality based on a multimodal large model according to claim 1, characterized in that: The method also includes a finite state machine, which manages the calling relationship and execution order between various intelligent agents according to preset various state definitions and state transfer conditions.

7. The method for detecting crop fruit quality based on a multimodal large model according to claim 1, characterized in that: The method also includes setting a global maximum number of retries. Each time the semantic segmentation agent or the semantic segmentation review agent is executed, a counter is incremented by one. When the counter reaches the global maximum number of retries, the semantically segmented crop fruit image in the current semantic segmentation agent is output.

8. A crop fruit quality detection system based on a multimodal large model, characterized in that: The invention comprises a multimodal large model established based on the LISA structure, which is used for detecting the quality of crop fruits; wherein, an image preprocessing agent is used to preprocess the input crop fruit image; the preprocessed crop fruit image is input into a semantic segmentation agent to extract the fruit area through semantic segmentation; the semantically segmented crop fruit image is input into a semantic segmentation review agent, and the semantic segmentation is reviewed by comparing the features of the actual crop fruit with the extracted fruit area; if the review result is that it is a fruit, the semantically segmented crop fruit image is input into a quality inspection agent, otherwise, it is returned to the semantic segmentation agent to re-semantically segment; the quality inspection agent detects the quality of the fruit area, and outputs the detection result and a natural language description of the detection basis, wherein the quality includes maturity and pests and diseases.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the method for detecting the quality of crop fruits based on a multimodal large model according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for detecting the quality of crop fruits based on a multimodal large model according to any one of claims 1 to 7 is implemented.

Images