Multi-modal large model-based scattered object influence analysis method in highway scene

Abstract

The invention discloses a throwing object influence analysis method based on a multi-mode large model in an expressway scene, and belongs to the technical field of intelligent traffic. In order to improve the real-time performance and accuracy of monitoring of the thrown objects, a segmentation module carries out fine-grained segmentation operation on image data output by a video acquisition and management module, and carries out segmentation and preliminary identification on the thrown objects in an image; the instruction generation module is used for integrating data from the instance segmentation module, the memory module and the thrown object detection task description, generating an output instruction and outputting the output instruction to the large model reasoning module for executing a thrown object analysis task; the input of the large model reasoning module comprises an output instruction of the instruction generation module and image data preprocessed by the video acquisition and management module, and the large model reasoning module is used for carrying out deep analysis and reasoning on the input data to generate an influence analysis result of the expressway thrown objects; and the evaluation module is used for comprehensively evaluating the expressway thrown object influence analysis result output by the large model reasoning module.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent transportation technology, and specifically relates to a method for analyzing the impact of spilled objects based on a multimodal large model in a highway scenario. Background Art

[0002] The spilled objects may include cargo, construction materials, parts that fall off the vehicle, etc., which may appear at any time in vehicles traveling at high speeds. Therefore, the detection, identification and impact analysis of spilled objects on highways are particularly important.

[0003] Highways are important transportation networks, and ensuring their safe and efficient passage is of utmost importance. On highways, spills may include cargo, waste, debris, etc. that fall from vehicles. In addition, cleaning up spills requires temporary lane closures, which slows down traffic flow, causes congestion, and affects highway traffic efficiency. Therefore, ensuring safe loading of cargo and proper disposal of waste are essential to maintaining the safety and smooth flow of highways.

[0004] Traditional spill monitoring mainly relies on manual inspections and fixed camera monitoring, which has problems such as delayed response and limited monitoring range. Summary of the invention

[0005] The problem to be solved by the present invention is to improve the real-time performance and accuracy of spillage monitoring, and propose a spillage impact analysis method based on a multimodal large model in a highway scenario.

[0006] To achieve the above object, the present invention is implemented through the following technical solutions:

[0007] A method for analyzing the impact of spilled objects on a highway based on a multimodal large model is implemented in a highway spilled object impact analysis system based on a multimodal large model. The highway spilled object impact analysis system based on a multimodal large model includes a video acquisition and management module, an instance segmentation module, a large model reasoning module, an instruction generation module, an evaluation module, a memory module, and a reflection module.

[0008] The video acquisition and management module is respectively connected to the instance segmentation module and the large model reasoning module, the instance segmentation module is connected to the instruction generation module, and the instruction generation module is respectively connected to the large model reasoning module and the memory module;

[0009] The large model reasoning module is connected to the evaluation module, the evaluation module is respectively connected to the memory module and the reflection module, and the reflection module is connected to the memory module;

[0010] The video acquisition and management module is used to acquire highway images and video data in real time and perform image preprocessing;

[0011] The instance segmentation module performs fine-grained segmentation operations on the image data output by the video acquisition and management module based on the visual segmentation basic model SAM of the Transformer architecture, and performs segmentation and preliminary recognition of the scattered objects in the image;

[0012] The instruction generation module integrates the data from the instance segmentation module, the memory module and the spilled object detection task description, generates output instructions and outputs them to the large model reasoning module for executing the spilled object analysis task;

[0013] The input of the large model reasoning module includes the output instructions of the instruction generation module and the image data preprocessed by the video acquisition and management module. The large model reasoning module is used to perform in-depth analysis and reasoning on the input data to generate the impact analysis results of the spilled objects on the highway;

[0014] The evaluation module is used to comprehensively evaluate the impact analysis results of highway spills output by the large model reasoning module;

[0015] The memory module is used to store experience data in historical scenes of highway spillage;

[0016] The steps include:

[0017] S1. The video acquisition and management module acquires highway images and video data in real time, performs image preprocessing, and outputs preprocessed image data;

[0018] S2. The instance segmentation module performs fine-grained segmentation operations on the preprocessed image data obtained in step S1 based on the visual segmentation basic model SAM of the Transformer architecture, segments and preliminarily identifies the scattered objects in the image, and obtains segmentation data and related metadata, wherein the metadata includes the area, shape, and position of the scattered objects;

[0019] S3. Input the segmented data and related metadata obtained in step S2, the experience data of the highway spilled objects historical scene in the memory module and the data describing the task of spilled objects detection to the instruction generation module, integrate them, and generate output instructions;

[0020] S4. Input the output instruction obtained in step S3 and the preprocessed image data obtained in step S1 to the large model reasoning module, perform in-depth analysis and reasoning, and generate the impact analysis results of the spilled objects on the highway;

[0021] S5. Input the impact analysis results of the highway spillage obtained in step S4 into the evaluation module, conduct a comprehensive evaluation, and obtain the evaluation results;

[0022] S6. Input the evaluation results obtained in step S5 into the memory module and the reflection module respectively, use the memory module to save the experience in historical scenarios, and use the reflection module to reflect and analyze the process of analyzing the impact of spilled objects on highways, and identify and correct potential problems.

[0023] Furthermore, the reflection module reflects on and analyzes the process of analyzing the impact of spilled materials on the highway, and identifies and corrects existing potential problems.

[0024] Furthermore, the specific implementation method of step S1 includes the following steps:

[0025] S1.1. Use interval acquisition strategy to collect video and collect highway images and video data in real time;

[0026] S1.2. For the highway image and video data collected in step S1.1, an adaptive filter is used to remove random noise in the image;

[0027] S1.3. The video data from which random noise is removed in step S1.2 is compressed using a video encoding algorithm, and metadata is added to the compressed video data and stored in a video acquisition and management module, wherein the metadata includes time, camera position, and shooting angle.

[0028] Furthermore, the interval acquisition strategy of step S1.1 is to set a 1s interval for video acquisition. When the network delay is greater than 5s, an adaptive acquisition strategy is used for acquisition.

[0029] Furthermore, the specific implementation method of step S2 includes the following steps:

[0030] S2.1. Preprocess the preprocessed image data obtained in step S1, and the calculation expression is:

[0031]

[0032] Among them, G(i,j) represents the edge strength of pixel (i,j), and Respectively represent the gradient of the image in the horizontal and vertical directions, which are used to detect the edge information of the scattered objects in the image;

[0033] S2.2. For the preprocessed image data obtained in step S2.1, set the candidate regions to be input into the visual segmentation basic model SAM, perform feature extraction and classification operations, and generate segmentation masks for each instance;

[0034] S2.3. post-processing the segmentation mask obtained in step S2.2, including edge smoothing, hole filling and region merging operations of the segmentation mask;

[0035] The calculation expression for edge smoothing is:

[0036]

[0037] Among them, Smooth(x,y) is the kernel function, σ is the standard deviation of the Gaussian function, and (x,y) is the position of the pixel;

[0038] Hole filling is used to fill the holes in the segmentation mask. The calculation expression is:

[0039]

[0040] Among them, (x, y) is the position of the pixel, M f (x, y) is the mask value after padding, M(x, y) is the original mask value, S is the neighborhood of the structure element, which is a 3*3 matrix, and (k, l) is the offset in the neighborhood calculation;

[0041] After the holes are filled, the feature similarity threshold is judged. If it is judged to be greater than the feature similarity threshold, it is output as the segmentation result of post-processing. If it is judged to be less than or equal to the feature similarity threshold, the region is merged and then post-processed again;

[0042] The calculation expression for region merging is:

[0043]

[0044] Wherein, μ1 and μ2 are the color mean of the first and second sprinkler regions, respectively; max_color_distance is the maximum distance in the color space; max_spatial_distance is the maximum distance in the color space; (x1, y1) and (x2, y2) are the centroid of the first and second sprinkler regions, respectively; α and β are the weighting coefficients of the first and second sprinkler regions, respectively; and α+β=1;

[0045] S2.4. Perform preliminary recognition on the post-processed segmentation results obtained in step S2.3 to obtain relevant metadata, including the area, shape, and location of the spilled object.

[0046] Furthermore, the specific implementation method of step S3 includes the following steps:

[0047] S3.1. Input the segmented data and related metadata obtained in step S2 to the instruction generation module, the experience data of the highway spilled objects historical scene of the memory module, and the data describing the task of detecting spilled objects;

[0048] The experience data of the highway spilled objects historical scenes in the memory module include the characteristics of different spilled objects, historical detection results and corresponding processing strategies, where the processing strategy refers to the management strategy of managers to deal with different spilled objects;

[0049] The spilled object detection task description includes analyzing the impact of spilled objects on each lane, preferentially detecting certain types of spilled objects, and processing specific road area operations. The spilled object detection task description provides a target orientation for instruction generation;

[0050] S3.2. Perform data parsing on the input data received in step S3.1, including analyzing the instance segmentation results, extracting relevant experience from the memory module, and understanding and decomposing the task description; describe the task in yaml format and use indentation to represent the hierarchical structure. The specific steps are as follows:

[0051] S3.2.1. Perform feature extraction on the instance segmentation results to generate a vector representation including information such as the shape, size, and position of the spilled object;

[0052] S3.2.2. Parse the description of the spill detection task and clarify the specific requirements and constraints of the task, including time requirements and processing priorities;

[0053] S3.2.3. By comparing with the small sample experience in the memory module, find historical data similar to the current detection object or scene, extract processing experience, and the calculation formula is as follows:

[0054]

[0055] Among them, q is the vector describing the current scene, j is the jth feature component in q, and k i is the i-th memory item in the memory module that is most similar to the current scene, Sim(q, k i ) is the cosine similarity between the current scene description and the memory scene description, k i，j is the i-th memory item in the memory module that is most similar to the i-th feature component of the current scene, and n is the total number of feature components;

[0056] S3.2.4. Integrate the features extracted from the instance segmentation result in step S3.2.1, the spilled object detection task description parsed in step S3.2.2, and the processing experience extracted in step S3.2.3 to generate output instructions. The calculation formula is as follows:

[0057] Prompt=G(F s , M e , T d )

[0058] Among them, Prompt is the generated standardized instruction, G is the rule or algorithm for instruction generation, and Fs The features extracted for instance segmentation results, M e For memory modules to process experience, T d Description of the spilled object detection task for parsing.

[0059] Furthermore, the specific implementation method of step S4 includes the following steps:

[0060] S4.1. Set the large model reasoning module to the GPT4 multimodal large model, input the output instruction obtained in step S3 and the preprocessed image data obtained in step S1, perform in-depth analysis and reasoning to generate the impact analysis result R of highway spills, and the calculation formula is as follows:

[0061] R = f(Image, Prompt)

[0062] Among them, f is the inference function of the large model, and Image is the original image;

[0063] S4.2. Set up a standardized interface for the large model inference module, supporting JSON and XML data formats.

[0064] Furthermore, the specific implementation method of step S5 includes the following steps:

[0065] S5.1. Verify the results of the highway spillage analysis obtained in step S4, including accuracy check and logical consistency check;

[0066] Accuracy check refers to verifying the accuracy of the current highway spillage impact analysis results by comparing them with the historical data stored in the memory module, and checking whether they are consistent with the analysis results of similar situations in the past;

[0067] Logical consistency check verifies the logical relationship in the results of highway spillage impact analysis to ensure that the results of highway spillage impact analysis and the generated decision recommendations are logically consistent and reasonable;

[0068] Get the result verification and consistency check result R check The expression is:

[0069]

[0070] Where N is the total number of spilled objects detected, is the prediction result of the large model analysis module for the i-th spilled object, and yi is the historical data corresponding to the i-th spilled object in the memory module; is the ith logical relationship in the analysis results of the impact of spilled objects on highways; iis the corresponding logical relationship in the generated decision suggestion; M is the number of logical relationships that need to be checked in the analysis results of the impact of highway spillage; is an indicator function, which is 1 when the results are consistent, otherwise 0; is a logic relationship check function, which returns 1 when the two are consistent, otherwise it returns 0; μ and ν are weight coefficients of the accuracy check and logic consistency check result scores respectively;

[0071] S5.2. Situational adaptability assessment: Based on the verification results obtained in step S5.1, the assessment module evaluates the adaptability of the highway spillage impact analysis results in combination with the actual situation of the current highway scenario, including environmental adaptability and risk;

[0072] Environmental adaptability refers to evaluating the applicability of analysis results and predicting the possible error range based on current environmental conditions such as weather and traffic flow;

[0073] Risk assessment is to reassess the risk of spilled materials, predict their impact range and possible consequences in the current scenario, and provide early warning information for different risk levels;

[0074] Get the situational adaptability evaluation result R eval The expression is:

[0075]

[0076] Where K is the number of current environmental conditions; is the spilled object under the kth environmental condition The result of the adaptability assessment; ri is the impact of the ith spill in the current scene; γ and λ are the weight coefficients of the environmental adaptability and risk estimation result scores respectively;

[0077] S5.3. Based on step S5.1 and step S5.2, a comprehensive evaluation formula is constructed to obtain the expression of the comprehensive evaluation result Score:

[0078] Score = α·R check +β·R evαl

[0079] Among them, α is the weight coefficient of the result verification and consistency check results, and β is the weight coefficient of the situational adaptability assessment results.

[0080] Furthermore, in step S6, the expression for reflecting and analyzing the highway spillage impact analysis process using the reflection module is:

[0081] Ref=LLM(Error(i),Context(i))

[0082] Among them, Ref is the improved reflection prompt, LLM is the model used to generate reflections and improvement suggestions, Error(i) is the error description of the i-th result, and Context(i) is the scene description when the error occurs.

[0083] Beneficial effects of the present invention:

[0084] The method for analyzing the impact of spilled objects in highway scenarios based on a multimodal large model described in the present invention can efficiently and automatically segment and analyze. The present invention uses a multimodal pre-trained large model to automatically segment and identify dynamic and static spilled objects in highway scenarios, thereby reducing dependence on manual labeling and maintenance costs, while significantly improving detection efficiency and accuracy.

[0085] The method for analyzing the impact of spilled objects based on a multimodal large model in a highway scenario described in the present invention has a collaborative working mechanism. The collaborative working mechanism of the instance segmentation model and the large model proposed in the present invention can accurately identify and analyze the nature of spilled objects and their impact on traffic in a complex highway environment by utilizing advanced segmentation technology and large model reasoning, and has a high degree of robustness.

[0086] The method for analyzing the impact of spilled objects on highways based on a multimodal large model described in the present invention can learn autonomously and continuously. By introducing an autonomous and continuous learning method based on evaluation and reflection, the system can self-optimize the detection effect, adapt to the needs of different scenarios, and achieve continuous improvement and optimization of highway spilled object detection and analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 It is a structural schematic diagram of a spillage impact analysis system based on a multi-modal large model in a highway scenario according to the present invention;

[0088] Figure 2 It is a flow chart of a method for analyzing the impact of spilled objects based on a multi-modal large model in a highway scenario according to the present invention;

[0089] Figure 3 It is a video acquisition flow chart of the present invention;

[0090] Figure 4 The example segmentation flow chart of the present invention;

[0091] Figure 5 A flow chart is generated for the instructions of the present invention. DETAILED DESCRIPTION

[0092] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0093] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0094] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 5 The detailed instructions are as follows:

[0095] Embodiment 1:

[0096] A method for analyzing the impact of spilled objects on a highway based on a multimodal large model is implemented in a highway spilled object impact analysis system based on a multimodal large model, including a video acquisition and management module 1, an instance segmentation module 2, a large model reasoning module 3, an instruction generation module 4, an evaluation module 5, a memory module 6, and a reflection module 7;

[0097] The video acquisition and management module 1 is respectively connected to the instance segmentation module 2 and the large model reasoning module 3, the instance segmentation module 2 is connected to the instruction generation module 4, and the instruction generation module 4 is respectively connected to the large model reasoning module 3 and the memory module 6;

[0098] The large model reasoning module 3 is connected to the evaluation module 5, the evaluation module 5 is respectively connected to the memory module 6 and the reflection module 7, and the reflection module 7 is connected to the memory module 6;

[0099] The video acquisition and management module 1 is used to acquire highway images and video data in real time and perform image preprocessing;

[0100] The instance segmentation module 2 performs fine-grained segmentation operations on the image data output by the video acquisition and management module 1 based on the visual segmentation basic model SAM of the Transformer architecture, and performs segmentation and preliminary recognition of the scattered objects in the image;

[0101] The instruction generation module 4 integrates the data from the instance segmentation module 2, the memory module 6 and the spilled object detection task description, generates an output instruction and outputs it to the large model reasoning module 3 for executing the spilled object analysis task;

[0102] The input of the large model reasoning module 3 includes the output instructions of the instruction generation module 4 and the image data preprocessed by the video acquisition and management module 1. The large model reasoning module 3 is used to perform in-depth analysis and reasoning on the input data to generate the impact analysis results of the spilled objects on the highway;

[0103] The evaluation module 5 is used to comprehensively evaluate the impact analysis results of highway spills output by the large model reasoning module 3;

[0104] The memory module 6 is used to store the experience data of the historical scenes of spilled objects on the highway;

[0105] The steps include:

[0106] S1. The video acquisition and management module acquires highway images and video data in real time, performs image preprocessing, and outputs preprocessed image data;

[0107] Furthermore, the specific implementation method of step S1 includes the following steps:

[0108] S1.1. Use interval acquisition strategy to collect video and collect highway images and video data in real time;

[0109] Furthermore, the interval acquisition strategy of step S1.1 is to set a 1s interval for video acquisition. When the network delay is greater than 5s, an adaptive acquisition strategy is used for acquisition.

[0110] S1.2. For the highway image and video data collected in step S1.1, an adaptive filter is used to remove random noise in the image;

[0111] S1.3. The video data obtained by removing random noise from the image in step S1.2 is compressed using a video encoding algorithm, and then metadata is added to the compressed video data and stored in the video acquisition and management module, wherein the metadata includes time, camera position, and shooting angle;

[0112] S2. The instance segmentation module performs fine-grained segmentation operations on the preprocessed image data obtained in step S1 based on the visual segmentation basic model SAM of the Transformer architecture, segments and preliminarily identifies the scattered objects in the image, and obtains segmentation data and related metadata, wherein the metadata includes the area, shape, and position of the scattered objects;

[0113] Furthermore, the reflection module 7 reflects and analyzes the highway spillage impact analysis process, identifies and corrects potential problems;

[0114] Furthermore, the specific implementation method of step S2 includes the following steps:

[0115] S2.1. Preprocess the preprocessed image data obtained in step S1, and the calculation expression is:

[0116]

[0117] Among them, G(i,j) represents the edge strength of pixel (i,j), and Respectively represent the gradient of the image in the horizontal and vertical directions, which are used to detect the edge information of the scattered objects in the image;

[0118] S2.2. For the preprocessed image data obtained in step S2.1, set the candidate regions to be input into the visual segmentation basic model SAM, perform feature extraction and classification operations, and generate segmentation masks for each instance;

[0119] S2.3. post-processing the segmentation mask obtained in step S2.2, including edge smoothing, hole filling and region merging operations of the segmentation mask;

[0120] The calculation expression for edge smoothing is:

[0121]

[0122] Among them, Smooth(x,y) is the kernel function, σ is the standard deviation of the Gaussian function, and (x,y) is the position of the pixel;

[0123] Hole filling is used to fill the holes in the segmentation mask. The calculation expression is:

[0124]

[0125] Among them, (x, y) is the position of the pixel, M f (x, y) is the mask value after padding, M(x, y) is the original mask value, S is the neighborhood of the structure element, which is a 3*3 matrix, and (k, l) is the offset in the neighborhood calculation;

[0126] Hole filling is used to fill the holes in the segmentation mask;

[0127] After the holes are filled, the feature similarity threshold is judged. If it is judged to be greater than the feature similarity threshold, it is output as the segmentation result of post-processing. If it is judged to be less than or equal to the feature similarity threshold, the region is merged and then post-processed again;

[0128] Region merging refers to comparing the feature similarity between the scattered objects to eliminate the defects in the segmentation process and generate a smoother and more continuous segmentation boundary. The feature similarity of the present invention is calculated by using color similarity and spatial adjacency;

[0129] The calculation expression for region merging is:

[0130]

[0131] Wherein, μ1 and μ2 are the color mean of the first and second sprinkler regions, respectively; max_color_distance is the maximum distance in the color space; max_spatial_distance is the maximum distance in the color space; (x1, y1) and (x2, y2) are the centroid of the first and second sprinkler regions, respectively; α and β are the weighting coefficients of the first and second sprinkler regions, respectively; and α+β=1;

[0132] S2.4. Perform preliminary recognition on the post-processed segmentation results obtained in step S2.3 to obtain relevant metadata, including the area, shape, and location of the spilled object.

[0133] S3. Input the segmented data and related metadata obtained in step S2, the experience data of the highway spilled objects historical scene in the memory module and the data describing the task of spilled objects detection to the instruction generation module, integrate them, and generate output instructions;

[0134] Furthermore, the specific implementation method of step S3 includes the following steps:

[0135] S3.1. Input the segmented data and related metadata obtained in step S2 to the instruction generation module, the experience data of the highway spilled objects historical scene of the memory module, and the data describing the task of detecting spilled objects;

[0136] The experience data of the highway spilled objects historical scenes in the memory module include the characteristics of different spilled objects, historical detection results and corresponding processing strategies, where the processing strategy refers to the management strategy of managers to deal with different spilled objects;

[0137] The spilled object detection task description includes analyzing the impact of spilled objects on each lane, preferentially detecting certain types of spilled objects, and processing specific road area operations. The spilled object detection task description provides a target orientation for instruction generation;

[0138] S3.2. Perform data parsing on the input data received in step S3.1, including analyzing the instance segmentation results, extracting relevant experience from the memory module, and understanding and decomposing the task description; describe the task in yaml format and use indentation to represent the hierarchical structure. The specific steps are as follows:

[0139] S3.2.1. Perform feature extraction on the instance segmentation results to generate a vector representation including information such as the shape, size, and position of the spilled object;

[0140] S3.2.2. Parse the description of the spill detection task and clarify the specific requirements and constraints of the task, including time requirements and processing priorities;

[0141] S3.2.3. By comparing with the small sample experience in the memory module, find historical data similar to the current detection object or scene, extract processing experience, and the calculation formula is as follows:

[0142]

[0143] Among them, q is the vector describing the current scene, j is the jth feature component in q, and k i is the i-th memory item in the memory module that is most similar to the current scene, Sim(q, k i ) is the cosine similarity between the current scene description and the memory scene description, k i，j is the i-th memory item in the memory module that is most similar to the i-th feature component of the current scene, and n is the total number of feature components;

[0144] The core task of the instruction generation module is to generate input instructions suitable for passing to the large model based on the information obtained from the analysis. This process requires the module to comprehensively consider the current detection object, task requirements and historical experience, and generate accurate large model input instructions through certain logical reasoning and algorithm calculations. The module has a built-in instruction generation rule library, which is based on a large amount of experimental data and expert knowledge. Through these rules, the module can generate optimal instructions in different situations.

[0145] S3.2.4. Integrate the features extracted from the instance segmentation result in step S3.2.1, the spilled object detection task description parsed in step S3.2.2, and the processing experience extracted in step S3.2.3 to generate output instructions. The calculation formula is as follows:

[0146] Prompt=G(F s , M e , T d )

[0147] Among them, Prompt is the generated standardized instruction, G is the rule or algorithm for instruction generation, and Fs The features extracted for instance segmentation results, M e For memory modules to process experience, T d Description of the spilled object detection task for parsing.

[0148] The final generated instructions are output to the large model inference module through a standardized interface. In order to ensure the executability of the instructions, the module will perform an integrity check on the instructions before output to ensure that the instruction format is correct and the parameters are complete.

[0149] S4. Input the output instruction obtained in step S3 and the preprocessed image data obtained in step S1 to the large model reasoning module, perform in-depth analysis and reasoning, and generate the impact analysis results of the spilled objects on the highway;

[0150] Furthermore, the specific implementation method of step S4 includes the following steps:

[0151] S4.1. Set the large model reasoning module to the GPT4 multimodal large model, input the output instruction obtained in step S3 and the preprocessed image data obtained in step S1, perform in-depth analysis and reasoning to generate the impact analysis result R of highway spills, and the calculation formula is as follows:

[0152] R = f(Image, Prompt)

[0153] Among them, f is the inference function of the large model, and Image is the original image;

[0154] S4.2. Set up a standardized interface for the large model inference module, supporting JSON and XML data formats.

[0155] S5. Input the impact analysis results of the highway spillage obtained in step S4 into the evaluation module, conduct a comprehensive evaluation, and obtain the evaluation results;

[0156] Furthermore, the specific implementation method of step S5 includes the following steps:

[0157] S5.1. Verify the results of the highway spillage analysis obtained in step S4, including accuracy check and logical consistency check;

[0158] Accuracy check refers to verifying the accuracy of the current highway spillage impact analysis results by comparing them with the historical data stored in the memory module, and checking whether they are consistent with the analysis results of similar situations in the past;

[0159] Logical consistency check verifies the logical relationship in the results of highway spillage impact analysis to ensure that the results of highway spillage impact analysis and the generated decision recommendations are logically consistent and reasonable;

[0160] Get the result verification and consistency check result R check The expression is:

[0161]

[0162] Where N is the total number of spilled objects detected, is the prediction result of the large model analysis module for the i-th spilled object, y i is the historical data corresponding to the i-th spilled object in the memory module; is the ith logical relationship in the analysis results of the impact of spilled objects on highways; i is the corresponding logical relationship in the generated decision suggestion; M is the number of logical relationships that need to be checked in the analysis results of the impact of highway spillage; is an indicator function, which is 1 when the results are consistent, otherwise 0; is a logic relationship check function, which returns 1 when the two are consistent, otherwise it returns 0; μ and ν are weight coefficients of the accuracy check and logic consistency check result scores respectively;

[0163] S5.2. Situational adaptability assessment: Based on the verification results obtained in step S5.1, the assessment module evaluates the adaptability of the highway spillage impact analysis results in combination with the actual situation of the current highway scenario, including environmental adaptability and risk;

[0164] Environmental adaptability refers to evaluating the applicability of analysis results and predicting the possible error range based on current environmental conditions such as weather and traffic flow;

[0165] Risk assessment is to reassess the risk of spilled materials, predict their impact range and possible consequences in the current scenario, and provide early warning information for different risk levels;

[0166] Get the situational adaptability evaluation result R eval The expression is:

[0167]

[0168] Where K is the number of current environmental conditions; is the spilled object under the kth environmental condition The results of the adaptability assessment; i is the impact degree of the i-th spilled object in the current scene; γ and λ are the weight coefficients of environmental adaptability and risk estimation result scores respectively;

[0169] S5.3. Based on step S5.1 and step S5.2, a comprehensive evaluation formula is constructed to obtain the expression of the comprehensive evaluation result Score:

[0170] Score = α·R check +β·R eval

[0171] Among them, α is the weight coefficient of the result verification and consistency check results, and β is the weight coefficient of the situation adaptability assessment results;

[0172] S6. Input the evaluation results obtained in step S5 into the memory module and the reflection module respectively, use the memory module to save the experience in the historical scene, and use the reflection module to reflect and analyze the process of highway spillage impact analysis, identify and correct potential problems;

[0173] Furthermore, in step S6, the expression for reflecting and analyzing the highway spillage impact analysis process using the reflection module is:

[0174] Ref=LLM(Error(i),Context(i))

[0175] Among them, Ref is the improved reflection prompt, LLM is the model used to generate reflections and improvement suggestions, Error(i) is the error description of the i-th result, and Context(i) is the scene description when the error occurs.

[0176] Furthermore, the results with higher scores from the evaluation module will be directly saved in the memory module, and will be used as a few-shot reference for decision-making when encountering similar situations. For the results with lower scores from the evaluation module, the reflection module will first use the LLM to determine the areas that need improvement based on the evaluation scores, and conduct self-reflection to improve decision-making accuracy and achieve self-continuous learning. The reflection prompt includes a description of the scenario where the decision error occurred and the original erroneous reasoning process. The reflection module requires the LLM to point out the cause of the error, give the correct reasoning and decision, and the strategy to avoid similar errors in the future.

[0177] Furthermore, the memory module saves the experience (memory) in the historical scene, which mainly consists of two parts: scene description and corresponding reasoning process. The scene description is used as the key of memory to realize the retrieval of similar memory. The reasoning process records the correct way to handle the scene, which is the value of memory. The implementation steps are as follows:

[0178] S6.1. Memory initialization: The memory module is initialized by selecting a small number of representative scenarios and manually listing the correct reasoning and decision-making processes in these scenarios as the initial memory. The formula is as follows:

[0179] Mem={(k1,v1),(k2,v2),…,(k n ,v n ),}

[0180] Among them, Mem is memory and experience, k i represents the vectorized representation of the scene description, v i Indicates the corresponding reasoning process;

[0181] S6.2. Recall: After each inference, a text description of the current spilled object detection and analysis scene will be obtained. Before making a decision, these scene descriptions will be converted into a vector, which will be used as a key to find similar scenes and their corresponding reasoning processes or memories in the memory module. The selected memory will be provided to the instruction generation module in the form of a small number of reference samples to assist the large model reasoning module in making more accurate reasoning and decisions in the current scenario. The retrieval algorithm used in the memory module is the K-NN algorithm, which can select the i elements that are most similar to a given query in large-scale data based on a given query. The implementation formula is as follows:

[0182] FewShot=Sim(q,k i )

[0183] Among them, FewShot is a small sample experience, q represents the vector representation of the current scene description, and k i represents the most similar i memory items, Sim is the cosine similarity between the current scene description and the memory scene description;

[0184] S6.3. Memory storage: When the agent makes correct inferences and decisions or reflects on incorrect inferences, it gains experience. The team uses the scene description as a key, forms a memory together with the inference process and saves it to the memory module. By using a vectorized database to save the results, it is easy to achieve efficient retrieval of similar scenes.

[0185] What are the key technical points and the points to be protected of the present invention?

[0186] (1) Automatic segmentation and recognition. The present invention discloses a dynamic and static spilled object recognition and detection device, system and method based on a multimodal large model in a highway scenario. The method of automatically segmenting spilled objects reduces the cost of manual labeling and maintenance, realizes the segmentation of various unknown types of spilled objects, and performs scene reasoning analysis through a large model, which has high robustness and high precision.

[0187] (2) Collaborative mechanism between instance segmentation and large model. The present invention proposes a collaborative working mechanism between the instance segmentation model and the large model. Through the close combination of the two, efficient collaborative processing is achieved in the identification and analysis of spilled objects, further improving the recognition accuracy of the system.

[0188] (3) Autonomous learning and optimization: The present invention proposes an autonomous continuous learning method based on evaluation and reflection. The system can automatically optimize the recognition and analysis process according to environmental changes and detection results to ensure the detection effect and stability in different highway scenarios.

[0189] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0190] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for analyzing the impact of spilled objects on highways based on a multimodal large model, characterized in that: The method is implemented in a highway spillage impact analysis system based on a multimodal large model, wherein the highway spillage impact analysis system based on a multimodal large model comprises a video acquisition and management module (1), an instance segmentation module (2), a large model reasoning module (3), an instruction generation module (4), an evaluation module (5), a memory module (6), and a reflection module (7); The video acquisition and management module (1) is respectively connected to the instance segmentation module (2) and the large model reasoning module (3); the instance segmentation module (2) is connected to the instruction generation module (4); and the instruction generation module (4) is respectively connected to the large model reasoning module (3) and the memory module (6); The large model reasoning module (3) is connected to the evaluation module (5), the evaluation module (5) is respectively connected to the memory module (6) and the reflection module (7), and the reflection module (7) is connected to the memory module (6); The video acquisition and management module (1) is used to acquire highway images and video data in real time and perform image preprocessing; The instance segmentation module (2) performs fine-grained segmentation operations on the image data output by the video acquisition and management module (1) based on the visual segmentation basic model SAM of the Transformer architecture, and performs segmentation and preliminary recognition of the scattered objects in the image; The instruction generation module (4) integrates the data from the instance segmentation module (2), the memory module (6) and the spilled object detection task description, generates an output instruction and outputs it to the large model reasoning module (3) for executing the spilled object analysis task; The input of the large model reasoning module (3) includes the output instructions of the instruction generation module (4) and the image data preprocessed by the video acquisition and management module (1), and the large model reasoning module (3) is used to perform in-depth analysis and reasoning on the input data to generate an analysis result of the impact of spilled objects on the highway; The evaluation module (5) is used to comprehensively evaluate the impact analysis results of highway spills output by the large model reasoning module (3); The memory module (6) is used to store experience data in historical scenes of highway spillage; The steps include: S1. The video acquisition and management module acquires highway images and video data in real time, performs image preprocessing, and outputs preprocessed image data; S2. The instance segmentation module performs fine-grained segmentation operations on the preprocessed image data obtained in step S1 based on the visual segmentation basic model SAM of the Transformer architecture, segments and preliminarily identifies the scattered objects in the image, and obtains segmentation data and related metadata, wherein the metadata includes the area, shape, and position of the scattered objects; S3. Input the segmented data and related metadata obtained in step S2, the experience data of the highway spilled objects historical scene in the memory module and the data describing the task of spilled objects detection to the instruction generation module, integrate them, and generate output instructions; S4. Input the output instruction obtained in step S3 and the preprocessed image data obtained in step S1 to the large model reasoning module, perform in-depth analysis and reasoning, and generate the impact analysis results of the spilled objects on the highway; S5. Input the impact analysis results of the highway spillage obtained in step S4 into the evaluation module, conduct a comprehensive evaluation, and obtain the evaluation results; S6. Input the evaluation results obtained in step S5 into the memory module and the reflection module respectively, use the memory module to save the experience in historical scenarios, and use the reflection module to reflect and analyze the process of analyzing the impact of spilled objects on highways, and identify and correct potential problems.

2. According to claim 1, a method for analyzing the impact of spilled objects based on a multimodal large model in a highway scenario is characterized in that: The reflection module (7) reflects on and analyzes the process of analyzing the impact of spilled materials on the highway, and identifies and corrects existing potential problems.

3. The highway spillage impact analysis method based on a multi-modal large model according to claim 2 is characterized in that: The specific implementation method of step S1 includes the following steps: S1.

1. Use interval acquisition strategy to collect video and collect highway images and video data in real time; S1.

2. For the highway image and video data collected in step S1.1, an adaptive filter is used to remove random noise in the image; S1.

3. The video data from which random noise is removed in step S1.2 is compressed using a video encoding algorithm, and metadata is added to the compressed video data and stored in a video acquisition and management module, wherein the metadata includes time, camera position, and shooting angle.

4. The highway spillage impact analysis method based on a multi-modal large model according to claim 3 is characterized in that: The interval acquisition strategy of step S1.1 is to set a 1s interval for video acquisition. When the network delay is greater than 5s, an adaptive acquisition strategy is used for acquisition.

5. The highway spillage impact analysis method based on a multi-modal large model according to claim 4 is characterized in that: The specific implementation method of step S2 includes the following steps: S2.

1. Preprocess the preprocessed image data obtained in step S1, and the calculation expression is: Among them, G(i,j) represents the edge strength of pixel (i,j), and Respectively represent the gradient of the image in the horizontal and vertical directions, which are used to detect the edge information of the scattered objects in the image; S2.

2. For the preprocessed image data obtained in step S2.1, set the candidate regions to be input into the visual segmentation basic model SAM, perform feature extraction and classification operations, and generate segmentation masks for each instance; S2.

3. post-processing the segmentation mask obtained in step S2.2, including edge smoothing, hole filling and region merging operations of the segmentation mask; The calculation expression for edge smoothing is: Among them, Smooth(x,y) is the kernel function, σ is the standard deviation of the Gaussian function, and (x,y) is the position of the pixel; Hole filling is used to fill the holes in the segmentation mask. The calculation expression is: Among them, (x, y) is the position of the pixel, M f (x, y) is the mask value after padding, M(x, y) is the original mask value, S is the neighborhood of the structure element, which is a 3*3 matrix, and (k, l) is the offset in the neighborhood calculation; After the holes are filled, the feature similarity threshold is judged. If it is judged to be greater than the feature similarity threshold, it is output as the segmentation result of post-processing. If it is judged to be less than or equal to the feature similarity threshold, the region is merged and then post-processed again; The calculation expression for region merging is: Wherein, μ1 and μ2 are the color mean of the first and second sprinkler regions, respectively; max_color_distance is the maximum distance in the color space; max_spatial_distance is the maximum distance in the color space; (x1, y1) and (x2, y2) are the centroid of the first and second sprinkler regions, respectively; α and β are the weighting coefficients of the first and second sprinkler regions, respectively; and α+β=1; S2.

4. Perform preliminary recognition on the post-processed segmentation results obtained in step S2.3 to obtain relevant metadata, including the area, shape, and location of the spilled object.

6. The highway spillage impact analysis method based on a multi-modal large model according to claim 5 is characterized in that: The specific implementation method of step S3 includes the following steps: S3.

1. Input the segmented data and related metadata obtained in step S2 to the instruction generation module, the experience data of the highway spilled objects historical scene of the memory module, and the data describing the task of detecting spilled objects; The experience data of the highway spilled objects historical scenes in the memory module include the characteristics of different spilled objects, historical detection results and corresponding processing strategies, where the processing strategy refers to the management strategy of managers to deal with different spilled objects; The spilled object detection task description includes analyzing the impact of spilled objects on each lane, preferentially detecting certain types of spilled objects, and processing specific road area operations. The spilled object detection task description provides a target orientation for instruction generation; S3.

2. Perform data parsing on the input data received in step S3.1, including analyzing the instance segmentation results, extracting relevant experience from the memory module, and understanding and decomposing the task description; describe the task in yaml format and use indentation to represent the hierarchical structure. The specific steps are as follows: S3.2.

1. Perform feature extraction on the instance segmentation results to generate a vector representation including information such as the shape, size, and position of the spilled object; S3.2.

2. Parse the description of the spill detection task and clarify the specific requirements and constraints of the task, including time requirements and processing priorities; S3.2.

3. By comparing with the small sample experience in the memory module, find historical data similar to the current detection object or scene, extract processing experience, and the calculation formula is as follows: Among them, q is the vector describing the current scene, j is the jth feature component in q, and k i is the i-th memory item in the memory module that is most similar to the current scene, Sim(q,k i ) is the cosine similarity between the current scene description and the memory scene description, k i,j is the i-th memory item in the memory module that is most similar to the j-th feature component of the current scene, and n is the total number of feature components; S3.2.

4. Integrate the features extracted from the instance segmentation result in step S3.2.1, the spilled object detection task description parsed in step S3.2.2, and the processing experience extracted in step S3.2.3 to generate output instructions. The calculation formula is as follows: Prompt=G(F s ,M e ,T d ) Among them, Prompt is the generated standardized instruction, G is the rule or algorithm for instruction generation, and F s The features extracted for instance segmentation results, M e For memory modules to process experience, T d Description of the spilled object detection task for parsing.

7. The highway spillage impact analysis method based on a multi-modal large model according to claim 6 is characterized in that: The specific implementation method of step S4 includes the following steps: S4.

1. Set the large model reasoning module to the GPT4 multimodal large model, input the output instruction obtained in step S3 and the preprocessed image data obtained in step S1, perform in-depth analysis and reasoning to generate the impact analysis result R of highway spills, and the calculation formula is as follows: R=f(Image,Prompt) Among them, f is the inference function of the large model, and Image is the original image; S4.

2. Set up a standardized interface for the large model inference module, supporting JSON and XML data formats.

8. The highway spillage impact analysis method based on a multi-modal large model according to claim 7 is characterized in that: The specific implementation method of step S5 includes the following steps: S5.

1. Verify the results of the highway spillage analysis obtained in step S4, including accuracy check and logical consistency check; Accuracy check refers to verifying the accuracy of the current highway spillage impact analysis results by comparing them with the historical data stored in the memory module, and checking whether they are consistent with the analysis results of similar situations in the past; Logical consistency check verifies the logical relationship in the results of highway spillage impact analysis to ensure that the results of highway spillage impact analysis and the generated decision recommendations are logically consistent and reasonable; Get the result verification and consistency check result R check The expression is: Where N is the total number of spilled objects detected, is the prediction result of the large model analysis module for the i-th spilled object, y i is the historical data corresponding to the i-th spilled object in the memory module; is the ith logical relationship in the analysis results of the impact of spilled objects on highways; i is the corresponding logical relationship in the generated decision suggestion; M is the number of logical relationships that need to be checked in the analysis results of the impact of highway spillage; is an indicator function, which is 1 when the results are consistent, otherwise 0; is a logic relationship check function, which returns 1 when the two are consistent, otherwise it returns 0; μ and ν are weight coefficients of the accuracy check and logic consistency check result scores respectively; S5.

2. Situational adaptability assessment: Based on the verification results obtained in step S5.1, the assessment module evaluates the adaptability of the highway spillage impact analysis results in combination with the actual situation of the current highway scenario, including environmental adaptability and risk; Environmental adaptability refers to evaluating the applicability of analysis results and predicting the possible error range based on current environmental conditions such as weather and traffic flow; Risk assessment is to reassess the risk of spilled materials, predict their impact range and possible consequences in the current scenario, and provide early warning information for different risk levels; Get the situational adaptability evaluation result R eval The expression is: Where K is the number of current environmental conditions; is the spilled object under the kth environmental condition The results of the adaptability assessment; i is the impact degree of the i-th spilled object in the current scene; γ and λ are the weight coefficients of environmental adaptability and risk estimation result scores respectively; S5.

3. Based on step S5.1 and step S5.2, a comprehensive evaluation formula is constructed to obtain the expression of the comprehensive evaluation result Score: Score=α·R check +β·R eval Among them, α is the weight coefficient of the result verification and consistency check results, and β is the weight coefficient of the situational adaptability assessment results.

9. The highway spillage impact analysis method based on a multi-modal large model according to claim 8 is characterized in that: In step S6, the reflection module is used to reflect and analyze the process of analyzing the impact of spilled objects on the highway: Ref=LLM(Error(i),Context(i)) Among them, Ref is the improved reflection prompt, LLM is the model used to generate reflections and improvement suggestions, Error(i) is the error description of the i-th result, and Context(i) is the scene description when the error occurs.

Images