A Method for Effect Analysis of Spillage Based on Multimodal Large Model in Highway Scenarios

Abstract

This invention discloses a method for analyzing the impact of spillage on highways based on a multimodal large model, belonging to the field of intelligent transportation technology. To improve the real-time performance and accuracy of spillage monitoring, the invention employs an instance segmentation module to perform fine-grained segmentation on the image data output by the video acquisition and management module, segmenting and initially identifying spillage in the images. An instruction generation module integrates data from the instance segmentation module, the memory module, and the spillage detection task description, generating output instructions that are sent to the large model inference module to execute the spillage analysis task. The input to the large model inference module includes the output instructions from the instruction generation module and the preprocessed image data from the video acquisition and management module. The large model inference module performs in-depth analysis and inference on the input data to generate the impact analysis results of highway spillage. An evaluation module comprehensively evaluates the highway spillage impact analysis results output by the large model inference module.

Description

Technical Field

[0001] This invention belongs to the field of intelligent transportation technology, specifically relating to a method for analyzing the impact of spilled materials based on a multimodal large model in highway scenarios. Background Technology

[0002] Spilled materials may include goods, building materials, and parts that have fallen off vehicles, and they can appear at any time in vehicles traveling at high speeds. Therefore, the detection, identification, and impact analysis of spilled materials on highways are particularly important.

[0003] Highways are vital transportation networks, and ensuring their safe and efficient passage is of paramount importance. Spillage on highways can include goods, waste, and debris that have fallen from vehicles. Furthermore, cleaning up spillage requires temporary lane closures, which slows traffic flow, causes congestion, and impacts highway efficiency. Therefore, ensuring the safe loading of goods and the proper disposal of waste are crucial for maintaining the safety and smooth operation of highways.

[0004] Traditional methods of monitoring spilled materials mainly rely on manual inspections and fixed camera monitoring, which have problems such as delayed response and limited monitoring range. Summary of the Invention

[0005] The problem this invention aims to solve is to improve the real-time performance and accuracy of spillage monitoring, and proposes a spillage impact analysis method based on a multimodal large model in highway scenarios.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for analyzing the impact of spillage on highways based on a multimodal large model is implemented in a highway spillage impact analysis system based on a multimodal large model. The highway spillage impact analysis system based on a multimodal large model includes a video acquisition and management module, an instance segmentation module, a large model reasoning module, an instruction generation module, an evaluation module, a memory module, and a reflection module.

[0008] The video acquisition and management module is connected to the instance segmentation module and the large model inference module, respectively. The instance segmentation module is connected to the instruction generation module, and the instruction generation module is connected to the large model inference module and the memory module, respectively.

[0009] The large model reasoning module is connected to the evaluation module, the evaluation module is connected to the memory module and the reflection module respectively, and the reflection module is connected to the memory module;

[0010] The video acquisition and management module is used to acquire highway images and video data in real time and perform image preprocessing;

[0011] The instance segmentation module performs fine-grained segmentation operations on the image data output by the video acquisition and management module based on the SAM visual segmentation basic model of the Transformer architecture, and performs segmentation and preliminary identification of the projectiles in the image.

[0012] The instruction generation module integrates the data from the instance segmentation module, the memory module, and the description of the spill detection task, and generates an output instruction to be output to the large model inference module for executing the spill analysis task.

[0013] The input to the large model inference module includes the output instructions of the instruction generation module and the preprocessed image data of the video acquisition and management module. The large model inference module is used to perform in-depth analysis and inference on the input data to generate the impact analysis results of highway spills.

[0014] The evaluation module is used to comprehensively evaluate the impact analysis results of highway spills output by the large model inference module.

[0015] The memory module is used to store experience data from historical scenarios of highway spillage.

[0016] Includes the following steps:

[0017] S1. The video acquisition and management module acquires highway images and video data in real time, performs image preprocessing, and outputs the preprocessed image data;

[0018] S2. The instance segmentation module performs fine-grained segmentation on the preprocessed image data obtained in step S1 based on the visual segmentation basic model SAM of the Transformer architecture, segments and initially identifies the objects in the image, and obtains segmentation data and related metadata, including the area, shape and position of the objects.

[0019] S3. Input the segmentation data and related metadata obtained in step S2, the experience data in the historical scene of highway spills in the memory module, and the data describing the spill detection task into the instruction generation module, integrate them, and generate output instructions.

[0020] S4. Input the output command obtained in step S3 and the preprocessed image data obtained in step S1 into the large model inference module, perform in-depth analysis and inference, and generate the impact analysis results of highway spills.

[0021] S5. Input the impact analysis results of highway spills obtained in step S4 into the evaluation module to conduct a comprehensive evaluation and obtain the evaluation results;

[0022] S6. Input the evaluation results obtained in step S5 into the memory module and the reflection module respectively. Use the memory module to save the experience in the historical scenario, and use the reflection module to reflect on and analyze the impact analysis process of highway spills, and identify and correct potential problems.

[0023] Furthermore, the reflection module reflects on and analyzes the impact analysis process of highway spills, identifying and correcting potential problems.

[0024] Furthermore, the specific implementation method of step S1 includes the following steps:

[0025] S1.1. An interval acquisition strategy is adopted for video acquisition, and highway images and video data are acquired in real time;

[0026] S1.2. For the highway image and video data acquired in step S1.1, an adaptive filter is used to remove random noise from the images;

[0027] S1.3. The video data from step S1.2, which was used to remove random noise from the image, is compressed using a video encoding algorithm. Then, metadata is added to the compressed video data and stored in the video acquisition and management module. The metadata includes time, camera position, and shooting angle.

[0028] Furthermore, the interval acquisition strategy in step S1.1 is to set a 1-second interval for video acquisition, and when the network latency is greater than 5 seconds, an adaptive acquisition strategy is adopted for acquisition.

[0029] Furthermore, the specific implementation method of step S2 includes the following steps:

[0030] S2.1. The preprocessed image data obtained in step S1 is preprocessed, and the calculation expression is as follows:

[0031]

[0032] Where G(i,j) represents the edge intensity of pixel (i,j). and These represent the gradients of the image in the horizontal and vertical directions, respectively, and are used to detect the edge information of the projectiles in the image;

[0033] S2.2. For the preprocessed image data obtained in step S2.1, set the candidate regions and input them into the visual segmentation basic model SAM to perform feature extraction and classification operations, and generate segmentation masks for each instance;

[0034] S2.3. Post-process the segmentation mask obtained in step S2.2, including edge smoothing, hole filling and region merging operations of the segmentation mask;

[0035] The expression for edge smoothing is:

[0036]

[0037] Where Smooth(x,y) is the kernel function, σ is the standard deviation of the Gaussian function, and (x,y) is the position of the pixel.

[0038] Hole filling is used to fill holes in a segmentation mask, and the calculation expression is:

[0039]

[0040] Where (x,y) represents the position of the pixel, M f (x,y) is the mask value after filling, M(x,y) is the original mask value, S is the neighborhood of the structuring element, which is a 3*3 matrix, and (k,l) is the offset in the neighborhood calculation.

[0041] After filling the holes, a feature similarity threshold is determined. If the feature similarity is greater than the threshold, the output is the segmentation result after post-processing. If the feature similarity is less than or equal to the threshold, the regions are merged and then post-processed again.

[0042] The calculation expression for region merging is:

[0043]

[0044] Where μ1 and μ2 are the color mean values ​​of the first and second sprayed object regions, respectively; max_color_distance is the maximum distance in the color space; max_spatial_distance is the maximum distance in the color space; (x1,y1) and (x2,y2) are the centroids of the first and second sprayed object regions, respectively; α and β are the weighting coefficients of the first and second sprayed object regions, respectively, and α+β=1.

[0045] S2.4. Perform preliminary identification on the post-processing segmentation results obtained in step S2.3 to obtain relevant metadata, including the area, shape, and location of the spilled material.

[0046] Furthermore, the specific implementation method of step S3 includes the following steps:

[0047] S3.1. Input the segmentation data and related metadata obtained in step S2 into the instruction generation module, the experience data in the historical scene of highway spills in the memory module, and the data describing the spill detection task.

[0048] The experience data in the historical scenarios of highway spills in the memory module includes the characteristics of different spills, historical detection results and corresponding handling strategies. The handling strategies refer to the management strategies of managers to deal with different spills.

[0049] The description of the spill detection task includes analyzing the impact of spills on each lane, prioritizing the detection of certain types of spills, and handling operations in specific road areas. The description of the spill detection task provides target guidance for instruction generation.

[0050] S3.2. Perform data parsing on the input data received in step S3.1, including analyzing the instance segmentation results, extracting relevant experience from the memory module, and understanding and decomposing the task description; describe the task using YAML format, and use indentation to represent the hierarchical structure. The specific steps are as follows:

[0051] S3.2.1. Extract features from the instance segmentation results to generate a vector representation including information such as the shape, size, and position of the projectile;

[0052] S3.2.2. Analyze the description of the spill detection task, clarify the specific requirements and constraints of the task, including time requirements and processing priorities;

[0053] S3.2.3. By comparing with the small sample experience in the memory module, find historical data similar to the current detection object or scene, extract processing experience, and calculate the following formula:

[0054]

[0055] Where q is the vector describing the current scene, j is the j-th feature component in q, and k i Sim(q, k) is the i-th memory item in the memory module that is most similar to the current scene. i Let k be the cosine similarity between the current scene description and the memory scene description. i，j Let be the i-th memory item in the memory module that is most similar to the i-th feature component of the current scene, where n is the total number of feature components;

[0056] S3.2.4. Integrate the features extracted from the instance segmentation results in step S3.2.1, the description of the debris detection task parsed in step S3.2.2, and the processing experience extracted in step S3.2.3 to generate output instructions. The calculation formula is as follows:

[0057] Prompt = G(F s M e T d )

[0058] Where Prompt represents the generated standardized instructions, G represents the rules or algorithm for instruction generation, and F represents the generated instructions.s M is the feature extracted from the instance segmentation results. e Based on memory module processing experience, T d This is a description of the task for parsing spill detection.

[0059] Furthermore, the specific implementation method of step S4 includes the following steps:

[0060] S4.1. Set the large model inference module to GPT4 multimodal large model, input the output command obtained in step S3 and the preprocessed image data obtained in step S1, perform deep analysis and inference to generate the impact analysis result R of highway debris, and the calculation formula is as follows:

[0061] R = f(Image, Prompt)

[0062] Where f is the inference function of the large model, and Image is the original image;

[0063] S4.2. Set up a standardized interface for the large model inference module, supporting JSON and XML data formats.

[0064] Furthermore, the specific implementation method of step S5 includes the following steps:

[0065] S5.1. Verify the impact analysis results of highway spills obtained in step S4, including accuracy checks and logical consistency checks;

[0066] Accuracy checks refer to verifying the accuracy of the current highway spill impact analysis results by comparing them with historical data stored in the memory module, and checking whether they are consistent with the analysis results of similar cases in the past.

[0067] The logical consistency check verifies the logical relationships in the impact analysis results of highway spills, ensuring that the impact analysis results and the generated decision recommendations are logically consistent and reasonable.

[0068] The result of the result verification and consistency check is obtained as R. check The expression is:

[0069]

[0070] Where N is the total number of scattered materials detected. yi represents the prediction result of the large model analysis module for the i-th scattering object, and yi represents the historical data corresponding to the i-th scattering object in the memory module. This is the i-th logical relation in the impact analysis results of highway spillage; z iM represents the corresponding logical relationships in the generated decision recommendations; M is the number of logical relationships that need to be checked in the impact analysis results of highway spills. This is an indicator function; it returns 1 if the results match, and 0 otherwise. This is a logical relationship check function that returns 1 if the two are consistent and 0 otherwise; μ and ν are the weighting coefficients for the accuracy check and logical consistency check scores, respectively.

[0071] S5.2. Context Adaptability Assessment: Based on the verification results obtained in step S5.1, the assessment module evaluates the adaptability of the impact analysis results of highway spills in combination with the actual situation of the current highway scenario, including environmental adaptability and risk.

[0072] Environmental adaptability refers to assessing the applicability of analysis results based on current environmental conditions such as weather and traffic flow, and predicting the possible range of error.

[0073] Risk assessment involves re-evaluating the risks of spilled materials, predicting their impact range and potential consequences in the current scenario, and providing early warning information for different risk levels.

[0074] The situational adaptability assessment result R was obtained. eval The expression is:

[0075]

[0076] Where K represents the number of current environmental conditions; To determine the effect of the spilled material under the k-th environmental condition The results of the adaptive assessment; ri represents the impact of the i-th spill in the current scenario; γ and λ are the weighting coefficients of the environmental adaptability and risk prediction scores, respectively;

[0077] S5.3. Based on steps S5.1 and S5.2, construct a comprehensive evaluation formula, and obtain the expression for the comprehensive evaluation result Score as follows:

[0078] Score = α·R check +β·R evαl

[0079] Where α is the weighting coefficient of the result verification and consistency check results, and β is the weighting coefficient of the situation adaptability assessment results.

[0080] Furthermore, the expression for reflecting on and analyzing the impact of highway spills using the reflection module in step S6 is as follows:

[0081] Ref=LLM(Error(i),Context(i))

[0082] Where Ref is the improved reflection prompt, LLM is the model used to generate reflection and improvement suggestions, Error(i) is the error description of the i-th result, and Context(i) is the scenario description when the error occurs.

[0083] The beneficial effects of this invention are:

[0084] This invention presents a method for analyzing the impact of spilled materials in highway scenarios based on a multimodal large model, enabling efficient and automated segmentation and analysis. By using a multimodal pre-trained large model, this invention automatically segments and identifies dynamic and static spilled materials in highway scenarios, reducing reliance on manual annotation, lowering maintenance costs, and significantly improving detection efficiency and accuracy.

[0085] This invention presents a method for analyzing the impact of spilled materials in highway scenarios based on a multimodal large model, featuring a collaborative working mechanism. The collaborative working mechanism between the instance segmentation model and the large model proposed in this invention, by utilizing advanced segmentation techniques and large model inference, can accurately identify and analyze the properties of spilled materials and their impact on traffic in complex highway environments, exhibiting high robustness.

[0086] The present invention discloses a method for analyzing the impact of spillage on highways based on a multimodal large model, which is capable of autonomous and continuous learning. By introducing an autonomous and continuous learning method based on evaluation and reflection, the system can self-optimize its detection performance, adapt to the needs of different scenarios, and achieve continuous improvement and optimization of the detection and analysis of spillage on highways. Attached Figure Description

[0087] Figure 1 This is a schematic diagram of the structure of a spill impact analysis system based on a multimodal large model in a highway scenario, as described in this invention.

[0088] Figure 2 This is a flowchart of a method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario, as described in this invention.

[0089] Figure 3 This is a flowchart of the video acquisition process of the present invention;

[0090] Figure 4 This is a flowchart illustrating the segmentation process of an example of the present invention;

[0091] Figure 5 A flowchart for generating instructions for this invention. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only for explaining the invention and are not intended to limit the invention; that is, the described specific embodiments are merely a part of the embodiments of the invention, and not all of them. The components of the specific embodiments of the invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations, and the invention may also have other embodiments.

[0093] Therefore, the following detailed description of specific embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected specific embodiments of the invention. All other specific embodiments obtained by those skilled in the art based on these specific embodiments without inventive effort are within the scope of protection of this invention.

[0094] To further understand the invention's content, features, and effects, the following specific embodiments are provided, along with accompanying drawings. Figure 1 -Appendix Figure 5 Detailed explanation is as follows:

[0095] Example 1:

[0096] A method for analyzing the impact of spillage on highways based on a multimodal large model is implemented in a highway spillage impact analysis system based on a multimodal large model, including a video acquisition and management module 1, an instance segmentation module 2, a large model reasoning module 3, an instruction generation module 4, an evaluation module 5, a memory module 6, and a reflection module 7.

[0097] The video acquisition and management module 1 is connected to the instance segmentation module 2 and the large model inference module 3 respectively. The instance segmentation module 2 is connected to the instruction generation module 4. The instruction generation module 4 is connected to the large model inference module 3 and the memory module 6 respectively.

[0098] The large model reasoning module 3 is connected to the evaluation module 5, the evaluation module 5 is connected to the memory module 6 and the reflection module 7 respectively, and the reflection module 7 is connected to the memory module 6;

[0099] The video acquisition and management module 1 is used to acquire highway images and video data in real time and perform image preprocessing;

[0100] The instance segmentation module 2, based on the Visual Segmentation Model (SAM) of the Transformer architecture, performs fine-grained segmentation operations on the image data output by the video acquisition and management module 1, and segments and preliminarily identifies the projectiles in the image.

[0101] The instruction generation module 4 integrates the data from the instance segmentation module 2, the memory module 6, and the description of the spill detection task, and generates an output instruction to be output to the large model inference module 3 for executing the spill analysis task.

[0102] The input of the large model inference module 3 includes the output instructions of the instruction generation module 4 and the preprocessed image data of the video acquisition and management module 1. The large model inference module 3 is used to perform in-depth analysis and inference on the input data to generate the impact analysis results of highway spills.

[0103] The evaluation module 5 is used to comprehensively evaluate the impact analysis results of highway spills output by the large model inference module 3.

[0104] The memory module 6 is used to store experience data from historical scenarios of highway spillage.

[0105] Includes the following steps:

[0106] S1. The video acquisition and management module acquires highway images and video data in real time, performs image preprocessing, and outputs the preprocessed image data;

[0107] Furthermore, the specific implementation method of step S1 includes the following steps:

[0108] S1.1. An interval acquisition strategy is adopted for video acquisition, and highway images and video data are acquired in real time;

[0109] Furthermore, the interval acquisition strategy in step S1.1 is to set a 1-second interval for video acquisition, and when the network latency is greater than 5 seconds, an adaptive acquisition strategy is adopted for acquisition.

[0110] S1.2. For the highway image and video data acquired in step S1.1, an adaptive filter is used to remove random noise from the images;

[0111] S1.3. The video data from which random noise was removed in step S1.2 is compressed using a video encoding algorithm. Then, metadata is added to the compressed video data and stored in the video acquisition and management module. The metadata includes time, camera position, and shooting angle.

[0112] S2. The instance segmentation module performs fine-grained segmentation on the preprocessed image data obtained in step S1 based on the visual segmentation basic model SAM of the Transformer architecture, segments and initially identifies the objects in the image, and obtains segmentation data and related metadata, including the area, shape and position of the objects.

[0113] Furthermore, the reflection module 7 reflects on and analyzes the impact analysis process of highway spills, identifying and correcting potential problems;

[0114] Furthermore, the specific implementation method of step S2 includes the following steps:

[0115] S2.1. The preprocessed image data obtained in step S1 is preprocessed, and the calculation expression is as follows:

[0116]

[0117] Where G(i,j) represents the edge intensity of pixel (i,j). and These represent the gradients of the image in the horizontal and vertical directions, respectively, and are used to detect the edge information of the projectiles in the image;

[0118] S2.2. For the preprocessed image data obtained in step S2.1, set the candidate regions and input them into the visual segmentation basic model SAM to perform feature extraction and classification operations, and generate segmentation masks for each instance;

[0119] S2.3. Post-process the segmentation mask obtained in step S2.2, including edge smoothing, hole filling and region merging operations of the segmentation mask;

[0120] The expression for edge smoothing is:

[0121]

[0122] Where Smooth(x,y) is the kernel function, σ is the standard deviation of the Gaussian function, and (x,y) is the position of the pixel.

[0123] Hole filling is used to fill holes in a segmentation mask, and the calculation expression is:

[0124]

[0125] Where (x,y) represents the position of the pixel, M f (x,y) is the mask value after filling, M(x,y) is the original mask value, S is the neighborhood of the structuring element, which is a 3*3 matrix, and (k,l) is the offset in the neighborhood calculation.

[0126] Hole filler is used to fill holes in a segmentation mask;

[0127] After filling the holes, a feature similarity threshold is determined. If the feature similarity is greater than the threshold, the output is the segmentation result after post-processing. If the feature similarity is less than or equal to the threshold, the regions are merged and then post-processed again.

[0128] Region merging refers to comparing the feature similarity between projectiles to eliminate imperfections in the segmentation process and generate smoother and more continuous segmentation boundaries. This invention calculates feature similarity using a combination of color similarity and spatial adjacency.

[0129] The calculation expression for region merging is:

[0130]

[0131] Where μ1 and μ2 are the color mean values ​​of the first and second sprayed object regions, respectively; max_color_distance is the maximum distance in the color space; max_spatial_distance is the maximum distance in the color space; (x1,y1) and (x2,y2) are the centroids of the first and second sprayed object regions, respectively; α and β are the weighting coefficients of the first and second sprayed object regions, respectively, and α+β=1.

[0132] S2.4. Perform preliminary identification on the post-processing segmentation results obtained in step S2.3 to obtain relevant metadata, including the area, shape, and location of the spilled material.

[0133] S3. Input the segmentation data and related metadata obtained in step S2, the experience data in the historical scene of highway spills in the memory module, and the data describing the spill detection task into the instruction generation module, integrate them, and generate output instructions.

[0134] Furthermore, the specific implementation method of step S3 includes the following steps:

[0135] S3.1. Input the segmentation data and related metadata obtained in step S2 into the instruction generation module, the experience data in the historical scene of highway spills in the memory module, and the data describing the spill detection task.

[0136] The experience data in the historical scenarios of highway spills in the memory module includes the characteristics of different spills, historical detection results and corresponding handling strategies. The handling strategies refer to the management strategies of managers to deal with different spills.

[0137] The description of the spill detection task includes analyzing the impact of spills on each lane, prioritizing the detection of certain types of spills, and handling operations in specific road areas. The description of the spill detection task provides target guidance for instruction generation.

[0138] S3.2. Perform data parsing on the input data received in step S3.1, including analyzing the instance segmentation results, extracting relevant experience from the memory module, and understanding and decomposing the task description; describe the task using YAML format, and use indentation to represent the hierarchical structure. The specific steps are as follows:

[0139] S3.2.1. Extract features from the instance segmentation results to generate a vector representation including information such as the shape, size, and position of the projectile;

[0140] S3.2.2. Analyze the description of the spill detection task, clarify the specific requirements and constraints of the task, including time requirements and processing priorities;

[0141] S3.2.3. By comparing with the small sample experience in the memory module, find historical data similar to the current detection object or scene, extract processing experience, and calculate the following formula:

[0142]

[0143] Where q is the vector describing the current scene, j is the j-th feature component in q, and k i Sim(q, k) is the i-th memory item in the memory module that is most similar to the current scene. i Let k be the cosine similarity between the current scene description and the memory scene description. i，j Let be the i-th memory item in the memory module that is most similar to the i-th feature component of the current scene, where n is the total number of feature components;

[0144] The core task of the instruction generation module is to generate suitable input instructions for the large model based on the parsed information. This process requires the module to comprehensively consider the current detection object, task requirements, and historical experience, and generate accurate input instructions for the large model through logical reasoning and algorithmic calculation. The module has a built-in instruction generation rule base, which is based on a large amount of experimental data and expert knowledge. Through these rules, the module can generate optimal instructions under different conditions.

[0145] S3.2.4. Integrate the features extracted from the instance segmentation results in step S3.2.1, the description of the debris detection task parsed in step S3.2.2, and the processing experience extracted in step S3.2.3 to generate output instructions. The calculation formula is as follows:

[0146] Prompt = G(F s M e T d )

[0147] Where Prompt represents the generated standardized instructions, G represents the rules or algorithm for instruction generation, and F represents the generated instructions.s M is the feature extracted from the instance segmentation results. e Based on memory module processing experience, T d This is a description of the task for parsing spill detection.

[0148] The generated instructions are then output to the large model inference module via a standardized interface. To ensure the executability of the instructions, the module performs an integrity check on the instructions before outputting them, ensuring that the instruction format is correct and the parameters are complete.

[0149] S4. Input the output command obtained in step S3 and the preprocessed image data obtained in step S1 into the large model inference module, perform in-depth analysis and inference, and generate the impact analysis results of highway spills.

[0150] Furthermore, the specific implementation method of step S4 includes the following steps:

[0151] S4.1. Set the large model inference module to GPT4 multimodal large model, input the output command obtained in step S3 and the preprocessed image data obtained in step S1, perform deep analysis and inference to generate the impact analysis result R of highway debris, and the calculation formula is as follows:

[0152] R = f(Image, Prompt)

[0153] Where f is the inference function of the large model, and Image is the original image;

[0154] S4.2. Set up a standardized interface for the large model inference module, supporting JSON and XML data formats.

[0155] S5. Input the impact analysis results of highway spills obtained in step S4 into the evaluation module to conduct a comprehensive evaluation and obtain the evaluation results;

[0156] Furthermore, the specific implementation method of step S5 includes the following steps:

[0157] S5.1. Verify the impact analysis results of highway spills obtained in step S4, including accuracy checks and logical consistency checks;

[0158] Accuracy checks refer to verifying the accuracy of the current highway spill impact analysis results by comparing them with historical data stored in the memory module, and checking whether they are consistent with the analysis results of similar cases in the past.

[0159] The logical consistency check verifies the logical relationships in the impact analysis results of highway spills, ensuring that the impact analysis results and the generated decision recommendations are logically consistent and reasonable.

[0160] The result of the result verification and consistency check is obtained as R. check The expression is:

[0161]

[0162] Where N is the total number of scattered materials detected. For the large model analysis module, y represents the prediction result of the i-th scattering object. i This refers to the historical data corresponding to the i-th scattered object in the memory module; This is the i-th logical relation in the impact analysis results of highway spillage; z i M represents the corresponding logical relationships in the generated decision recommendations; M is the number of logical relationships that need to be checked in the impact analysis results of highway spills. This is an indicator function; it returns 1 if the results match, and 0 otherwise. This is a logical relationship check function that returns 1 if the two are consistent and 0 otherwise; μ and ν are the weighting coefficients for the accuracy check and logical consistency check scores, respectively.

[0163] S5.2. Context Adaptability Assessment: Based on the verification results obtained in step S5.1, the assessment module evaluates the adaptability of the impact analysis results of highway spills in combination with the actual situation of the current highway scenario, including environmental adaptability and risk.

[0164] Environmental adaptability refers to assessing the applicability of analysis results based on current environmental conditions such as weather and traffic flow, and predicting the possible range of error.

[0165] Risk assessment involves re-evaluating the risks of spilled materials, predicting their impact range and potential consequences in the current scenario, and providing early warning information for different risk levels.

[0166] The situational adaptability assessment result R was obtained. eval The expression is:

[0167]

[0168] Where K represents the number of current environmental conditions; To determine the effect of the spilled material under the k-th environmental condition The results of the adaptation assessment; r i γ represents the impact of the i-th spill in the current scenario; γ and λ are the weighting coefficients for the environmental adaptability and risk prediction scores, respectively.

[0169] S5.3. Based on steps S5.1 and S5.2, construct a comprehensive evaluation formula, and obtain the expression for the comprehensive evaluation result Score as follows:

[0170] Score = α·R check +β·R eval

[0171] Where α is the weighting coefficient of the result verification and consistency check result, and β is the weighting coefficient of the situation adaptability assessment result.

[0172] S6. Input the evaluation results obtained in step S5 into the memory module and the reflection module respectively. Use the memory module to save the experience in the historical scenario, and use the reflection module to reflect on and analyze the impact analysis process of highway spills, and identify and correct potential problems.

[0173] Furthermore, the expression for reflecting on and analyzing the impact of highway spills using the reflection module in step S6 is as follows:

[0174] Ref=LLM(Error(i),Context(i))

[0175] Where Ref is the improved reflection prompt, LLM is the model used to generate reflection and improvement suggestions, Error(i) is the error description of the i-th result, and Context(i) is the scenario description when the error occurs.

[0176] Furthermore, the reflection module directly saves results with high scores from the evaluation module to the memory module, serving as a few-shot reference for decision-making in similar situations. For results with low scores from the evaluation module, the reflection module first uses LLM to identify areas for improvement based on the evaluation scores, then engages in self-reflection to improve decision-making accuracy and achieve continuous self-learning. The reflection prompts include a description of the scenario in which the decision-making error occurred and the original reasoning process behind the error. The reflection module requires LLM to identify the cause of the error, provide the correct reasoning and decision, and strategies to avoid similar errors in the future.

[0177] Furthermore, the memory module stores experiences (memories) from historical scenarios, mainly consisting of two parts: a scenario description and the corresponding reasoning process. The scenario description serves as the key for the memory to retrieve similar memories. The reasoning process records the correct method for handling the scenario and is the value of the memory. The implementation steps are as follows:

[0178] S6.1. Memory Initialization: Memory module initialization involves selecting a small number of representative scenarios and manually listing the correct reasoning and decision-making processes in these scenarios as initial memories, using the following formula:

[0179] Mem={(k1,v1),(k2,v2),…,(k n ,v n ),}

[0180] Among them, Mem represents memory and experience, and k i The vectorized representation of the scene description, v i This indicates the corresponding reasoning process;

[0181] S6.2. Recall: After each inference, a textual description of the current debris detection and analysis scenario is obtained. Before making a decision, these scenario descriptions are transformed into a vector, which serves as the key to search for similar scenarios and their corresponding inference processes or memories in the memory module. The selected memories are provided to the instruction generation module as a small number of reference samples to assist the large model inference module in making more accurate inferences and decisions in the current scenario. The retrieval algorithm used in the memory module is the K-NN algorithm, which can select the i elements most similar to a given query from a large-scale dataset, as shown in the following formula:

[0182] FewShot=Sim(q,k i )

[0183] Where FewShot is the few-sample experience, q represents the vector representation of the current scene description, and k i Let i represent the i most similar memory items, and Sim be the cosine similarity between the current scene description and the memory scene description;

[0184] S6.3. Memory Storage: When an agent makes correct reasoning and decisions or reflects on incorrect reasoning processes, it gains experience. The team uses scene descriptions as keys, forming memories along with the reasoning process and storing them in the memory module. Using a vectorized database to store results facilitates efficient similar scene retrieval.

[0185] What are the key technical points and protected aspects of this invention?

[0186] (1) Automatic segmentation and recognition. This invention discloses a dynamic and static debris recognition and detection device, system and method based on a multimodal large model in highway scenarios. By automatically segmenting debris, the manual annotation and maintenance costs are reduced, and the segmentation of various unknown types of debris is achieved. Scene reasoning analysis is performed through a large model, which has high robustness and high accuracy.

[0187] (2) Instance segmentation and large model collaboration mechanism. This invention proposes a collaborative working mechanism between the instance segmentation model and the large model. Through the close integration of the two, efficient collaborative processing is achieved in the identification and analysis of projectiles, further improving the system's identification accuracy.

[0188] (3) Autonomous learning and optimization: This invention proposes an autonomous and continuous learning method based on evaluation and reflection. The system can automatically optimize the identification and analysis process according to environmental changes and detection results, ensuring the detection effect and stability in different highway scenarios.

[0189] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0190] Although this application has been described above with reference to specific embodiments, various modifications can be made and components can be replaced with equivalents without departing from the scope of this application. In particular, as long as there is no structural conflict, the features in the specific embodiments disclosed in this application can be combined with each other in any way. The lack of an exhaustive description of these combinations in this specification is merely for the sake of brevity and resource conservation. Therefore, this application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario, characterized in that, This is implemented in a highway spill impact analysis system based on a multimodal large model. The highway spill impact analysis system based on a multimodal large model includes a video acquisition and management module (1), an instance segmentation module (2), a large model reasoning module (3), an instruction generation module (4), an evaluation module (5), a memory module (6), and a reflection module (7). The video acquisition and management module (1) is connected to the instance segmentation module (2) and the large model inference module (3) respectively. The instance segmentation module (2) is connected to the instruction generation module (4). The instruction generation module (4) is connected to the large model inference module (3) and the memory module (6) respectively. The large model reasoning module (3) is connected to the evaluation module (5), the evaluation module (5) is connected to the memory module (6) and the reflection module (7) respectively, and the reflection module (7) is connected to the memory module (6); The video acquisition and management module (1) is used to acquire highway images and video data in real time and perform image preprocessing; The instance segmentation module (2) performs fine-grained segmentation operations on the image data output by the video acquisition and management module (1) based on the visual segmentation basic model SAM of the Transformer architecture, and performs segmentation and preliminary identification of the objects in the image; The instruction generation module (4) integrates the data from the instance segmentation module (2), the memory module (6) and the description of the spill detection task, and generates an output instruction to be output to the large model inference module (3) for executing the spill analysis task; The input of the large model reasoning module (3) includes the output instructions of the instruction generation module (4) and the preprocessed image data of the video acquisition and management module (1). The large model reasoning module (3) is used to perform in-depth analysis and reasoning on the input data to generate the impact analysis results of highway spills. The evaluation module (5) is used to comprehensively evaluate the impact analysis results of highway spills output by the large model inference module (3); The memory module (6) is used to store experience data from historical scenarios of highway spillage. Includes the following steps: S1. The video acquisition and management module acquires highway images and video data in real time, performs image preprocessing, and outputs the preprocessed image data; S2. The instance segmentation module performs fine-grained segmentation on the preprocessed image data obtained in step S1 based on the visual segmentation basic model SAM of the Transformer architecture, segments and initially identifies the objects in the image, and obtains segmentation data and related metadata, including the area, shape and position of the objects. S3. Input the segmentation data and related metadata obtained in step S2, the experience data in the historical scene of highway spills in the memory module, and the data describing the spill detection task into the instruction generation module, integrate them, and generate output instructions. S4. Input the output command obtained in step S3 and the preprocessed image data obtained in step S1 into the large model inference module, perform in-depth analysis and inference, and generate the impact analysis results of highway spills. S5. Input the impact analysis results of highway spills obtained in step S4 into the evaluation module to conduct a comprehensive evaluation and obtain the evaluation results; S6. Input the evaluation results obtained in step S5 into the memory module and the reflection module respectively. Use the memory module to save the experience in the historical scenario, and use the reflection module to reflect on and analyze the impact analysis process of highway spills, and identify and correct potential problems.

2. The method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario according to claim 1, characterized in that, The reflection module (7) reflects on and analyzes the impact analysis process of highway spills, and identifies and corrects potential problems.

3. The method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario according to claim 2, characterized in that, The specific implementation method of step S1 includes the following steps: S1.

1. An interval acquisition strategy is adopted for video acquisition, and highway images and video data are acquired in real time; S1.

2. For the highway image and video data acquired in step S1.1, an adaptive filter is used to remove random noise from the images; S1.

3. The video data from step S1.2, which was used to remove random noise from the image, is compressed using a video encoding algorithm. Then, metadata is added to the compressed video data and stored in the video acquisition and management module. The metadata includes time, camera position, and shooting angle.

4. The method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario according to claim 3, characterized in that, The interval acquisition strategy in step S1.1 is to set a 1-second interval for video acquisition. When the network latency is greater than 5 seconds, an adaptive acquisition strategy is adopted for acquisition.

5. The method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario according to claim 4, characterized in that, The specific implementation method of step S2 includes the following steps: S2.

1. The preprocessed image data obtained in step S1 is preprocessed, and the calculation expression is as follows: Where G(i,j) represents the edge intensity of pixel (i,j). and These represent the gradients of the image in the horizontal and vertical directions, respectively, and are used to detect the edge information of the projectiles in the image; S2.

2. For the preprocessed image data obtained in step S2.1, set the candidate regions and input them into the visual segmentation basic model SAM to perform feature extraction and classification operations, and generate segmentation masks for each instance; S2.

3. Post-process the segmentation mask obtained in step S2.2, including edge smoothing, hole filling and region merging operations of the segmentation mask; The expression for edge smoothing is: Where Smooth(x,y) is the kernel function, σ is the standard deviation of the Gaussian function, and (x,y) is the position of the pixel. Hole filling is used to fill holes in a segmentation mask, and the calculation expression is: Where (x,y) represents the position of the pixel, M f (x,y) is the mask value after filling, M(x,y) is the original mask value, S is the neighborhood of the structuring element, which is a 3*3 matrix, and (k,l) is the offset in the neighborhood calculation. After filling the holes, a feature similarity threshold is determined. If the feature similarity is greater than the threshold, the output is the segmentation result after post-processing. If the feature similarity is less than or equal to the threshold, the regions are merged and then post-processed again. The calculation expression for region merging is: Where μ1 and μ2 are the color mean values ​​of the first and second sprayed object regions, respectively; max_color_distance is the maximum distance in the color space; max_spatial_distance is the maximum distance in the color space; (x1,y1) and (x2,y2) are the centroids of the first and second sprayed object regions, respectively; α and β are the weighting coefficients of the first and second sprayed object regions, respectively, and α+β=1. S2.

4. Perform preliminary identification on the post-processing segmentation results obtained in step S2.3 to obtain relevant metadata, including the area, shape, and location of the spilled material.

6. The method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario according to claim 5, characterized in that, The specific implementation method of step S3 includes the following steps: S3.

1. Input the segmentation data and related metadata obtained in step S2 into the instruction generation module, the experience data in the historical scene of highway spills in the memory module, and the data describing the spill detection task. The experience data in the historical scenarios of highway spills in the memory module includes the characteristics of different spills, historical detection results and corresponding handling strategies. The handling strategies refer to the management strategies that managers use to deal with different spills. The description of the spill detection task includes analyzing the impact of spills on each lane, prioritizing the detection of certain types of spills, and handling operations in specific road areas. The description of the spill detection task provides target guidance for instruction generation. S3.

2. Perform data parsing on the input data received in step S3.1, including analyzing the instance segmentation results, extracting relevant experience from the memory module, and understanding and decomposing the task description; describe the task using YAML format, and use indentation to represent the hierarchical structure. The specific steps are as follows: S3.2.

1. Extract features from the instance segmentation results to generate a vector representation including information such as the shape, size, and position of the projectile; S3.2.

2. Analyze the description of the spill detection task, clarify the specific requirements and constraints of the task, including time requirements and processing priorities; S3.2.

3. By comparing with the small sample experience in the memory module, find historical data similar to the current detection object or scene, extract processing experience, and calculate the following formula: Where q is the vector describing the current scene, j is the j-th feature component in q, and k i Sim(q,k) is the i-th memory item in the memory module that is most similar to the current scene. i Let k be the cosine similarity between the current scene description and the memory scene description. i,j Let be the i-th memory entry in the memory module that is most similar to the j-th feature component of the current scene, and n be the total number of feature components; S3.2.

4. Integrate the features extracted from the instance segmentation results in step S3.2.1, the description of the debris detection task parsed in step S3.2.2, and the processing experience extracted in step S3.2.3 to generate output instructions. The calculation formula is as follows: Prompt=G(F s ,M e ,T d ) Where Prompt represents the generated standardized instructions, G represents the rules or algorithm for instruction generation, and F represents the generated instructions. s M is the feature extracted from the instance segmentation results. e Based on memory module processing experience, T d This is a description of the task for parsing spill detection.

7. The method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario according to claim 6, characterized in that, The specific implementation method of step S4 includes the following steps: S4.

1. Set the large model inference module to GPT4 multimodal large model, input the output command obtained in step S3 and the preprocessed image data obtained in step S1, perform deep analysis and inference to generate the impact analysis result R of highway debris, and the calculation formula is as follows: R = f(Image, Prompt) Where f is the inference function of the large model, and Image is the original image; S4.

2. Set up a standardized interface for the large model inference module, supporting JSON and XML data formats.

8. The method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario according to claim 7, characterized in that, The specific implementation method of step S5 includes the following steps: S5.

1. Verify the impact analysis results of highway spills obtained in step S4, including accuracy checks and logical consistency checks; Accuracy checks refer to verifying the accuracy of the current highway spill impact analysis results by comparing them with historical data stored in the memory module, and checking whether they are consistent with the analysis results of similar cases in the past. The logical consistency check verifies the logical relationships in the impact analysis results of highway spills, ensuring that the impact analysis results and the generated decision recommendations are logically consistent and reasonable. The result of the result verification and consistency check is obtained as R. check The expression is: Where N is the total number of scattered materials detected. For the large model analysis module, y represents the prediction result of the i-th scattering object. i This refers to the historical data corresponding to the i-th scattered object in the memory module; This is the i-th logical relation in the impact analysis results of highway spillage; z i M represents the corresponding logical relationships in the generated decision recommendations; M is the number of logical relationships that need to be checked in the impact analysis results of highway spills. This is an indicator function; it returns 1 if the results match, and 0 otherwise. This is a logical relationship check function that returns 1 if the two are consistent and 0 otherwise; μ and v are the weighting coefficients for the accuracy check and logical consistency check scores, respectively. S5.

2. Context Adaptability Assessment: Based on the verification results obtained in step S5.1, the assessment module evaluates the adaptability of the impact analysis results of highway spills in combination with the actual situation of the current highway scenario, including environmental adaptability and risk. Environmental adaptability refers to assessing the applicability of analysis results based on current environmental conditions such as weather and traffic flow, and predicting the possible range of error. Risk assessment involves re-evaluating the risks of spilled materials, predicting their impact range and potential consequences in the current scenario, and providing early warning information for different risk levels. The situational adaptability assessment result R was obtained. eval The expression is: Where K represents the number of current environmental conditions; To determine the spillage under the k-th environmental condition The results of the adaptation assessment; r i γ represents the impact of the i-th spill in the current scenario; γ and λ are the weighting coefficients for the environmental adaptability and risk prediction scores, respectively. S5.

3. Based on steps S5.1 and S5.2, construct a comprehensive evaluation formula, and obtain the expression for the comprehensive evaluation result Score as follows: Score=α·R check +β·R eval Where α is the weighting coefficient of the result verification and consistency check results, and β is the weighting coefficient of the situation adaptability assessment results.

9. The method for analyzing the impact of spilled materials based on a multimodal large model in a highway scenario according to claim 8, characterized in that, The expression for the reflection and analysis process of the highway spill impact analysis using the reflection module in step S6 is as follows: Ref=LLM(Error(i),Context(i)) Where Ref is the improved reflection prompt, LLM is the model used to generate reflection and improvement suggestions, Error(i) is the error description of the i-th result, and Context(i) is the scenario description when the error occurs.

Images