Inspection diagnosis method and system based on local feature enhancement and heterogeneous large model

Abstract

The application discloses a method and system for inspection diagnosis based on local feature enhancement and a heterogeneous large model, which comprises the following steps: receiving a stream of inspection images collected by a plurality of terminals in real time, performing color space alignment and adaptive format preprocessing, and constructing an image sequence to be analyzed; performing first-round concurrent reasoning on the image sequence, performing global coarse screening and classification of device types, and obtaining a locked target; separating and enhancing the high-definition local feature map of a small key component through a bounding rectangle space clustering sorting and threshold interpolation amplification algorithm; constructing a local visual large model and an OCR dual-engine cooperative recognition mechanism to perform parallel feature extraction on the amplified high-definition local feature map, and obtaining a repair type feature set; establishing a prior feature injection mechanism, outputting an abnormal hidden danger list according to the repair type feature set, and accurately positioning and adaptively amplifying a small key area in a panoramic image through an HSV color space mask and a morphological algorithm, thereby effectively solving the visual illusion problem of a large model for small features.

Description

Technical Field

[0001] This invention relates to the field of intelligent industrial inspection technology, specifically to an inspection and diagnosis method and system based on local feature enhancement and heterogeneous large models. Background Technology

[0002] With the rapid development of the Industrial Internet of Things (IIoT) and robotics, automated inspections using mobile terminals such as robot dogs and drones have become a trend in complex industrial sites like water conservancy hubs, such as reservoir pumping stations and substations. These inspection terminals generate massive amounts of multimodal data daily, which is of great value for equipment status monitoring, fault early warning, and safe production. However, traditional industrial vision analysis heavily relies on customized small-model algorithms with weak generalization capabilities, and current mainstream inspection systems still face the following technical bottlenecks: 1. The contradiction between computing power and resolution in global view and minute feature analysis: The images collected by the inspection robot are mostly panoramic wide-format images that include the entire equipment. Key status indicators in the images, such as LCD screen values, dashboards, and indicator lights, occupy a very small area. If the entire image is directly input into the multimodal large model (VLM), the model is prone to visual illusions due to image token compression, making it impossible to accurately read minute values.

[0003] 2. Non-standard structured output of large models can easily lead to the collapse of automated pipelines: Industrial inspection systems require high-precision structured data output. However, in actual inference, large models often experience problems such as output truncation, redundant explanatory text, or unclosed brackets due to token restrictions or network fluctuations, which can directly interrupt subsequent program parsing.

[0004] 3. Single technology struggles to balance accurate recognition and global logical analysis: Traditional OCR (Optical Character Recognition) technology can extract numbers, but it lacks the ability to understand context such as the overall appearance of the device and thermal imaging temperature; while multimodal large models have the ability to understand, but they are not as sensitive to tiny numbers as OCR and lack an efficient heterogeneous engine collaborative scheduling framework.

[0005] 4. Low level of automation in report generation and lack of closed-loop diagnosis: Existing inspection systems mostly remain at the stage of discovering anomalies and capturing images. Maintenance personnel still need to spend a lot of effort manually compiling standardized inspection reports with pictures and text.

[0006] 5. The monitoring system lacks a long-term memory and prior experience accumulation mechanism: The data generated by traditional monitoring and inspection are mostly information silos, lacking a closed-loop mechanism to transform the equipment characteristics, environmental parameters and diagnostic conclusions of each inspection into system memory. Maintenance personnel cannot perform intelligent retrieval and traceability of the long-term historical status evolution of equipment based on natural language. Summary of the Invention

[0007] The present invention proposes an inspection and diagnosis method and system based on local feature enhancement and heterogeneous large models to solve the problems mentioned above in the background.

[0008] To achieve the above objectives, the present invention adopts the following technical solution: The inspection and diagnosis method based on local feature enhancement and heterogeneous large models of the present invention includes the following steps: S1. Receive inspection image streams collected from multiple sources in real time, perform color space alignment and adaptive format preprocessing, and construct the image sequence to be analyzed; S2. Based on the multimodal visual large model, the image sequence is subjected to the first round of concurrent inference, and global coarse screening and classification of device types are performed to obtain the locked target; S3. For the locked target, an adaptive region of interest extraction mechanism based on HSV color space mask and morphological closing operation is established. Through the bounding rectangle space clustering sorting and threshold interpolation amplification algorithm, high-definition local feature maps of small key components are separated and enhanced. S4. Construct a local visual large model and OCR dual-engine collaborative recognition mechanism to extract features in parallel from the magnified high-definition local feature map, and establish a dynamic regular truncation self-repair model for the non-standard JSON output of the large model. Through reverse key-value pair matching and automatic completion of missing fields, a repaired feature set is obtained. S5. Establish a prior feature injection mechanism to drive the large model to perform hallucination-free secondary joint reasoning based on the repair feature set and output an abnormal potential list; S6. Based on the temporal features and list of abnormal hazards in the high-definition local feature map, input the data into the large language model to perform a global logic chain in-depth diagnosis and adaptively generate a water conservancy inspection report.

[0009] Preferably, S1 includes: S11. Establish an asynchronous image stream monitoring mechanism for the complex network environment of water conservancy hubs, to receive visible light and infrared thermal imaging dual-light data transmitted back by the robot dog in real time, and to receive the original image sequence transmitted back by the inspection robot. ; Detecting the tensor dimension of an image If color channel dimensions Establish a color space mapping matrix Force mapping of single-channel grayscale or four-channel RGBA to a standard BGR format matrix :

[0010] S12. Construct an image tensor buffer and assign it to the image based on the timestamp. The data is serialized and encapsulated, stored in the queue to be analyzed, and then handed over to the downstream asynchronous thread pool for processing.

[0011] Preferably, S2 includes: S21. Let the first... The features of a frame panoramic image are The basic category suggestion keywords are The probability of the target device's existence is calculated using a multimodal large model:

[0012] S22. Establish a set of target category identifiers If the set of device type categories parsed by the large model is consistent with... The intersection of the two sets of elements is not empty, which means that the condition is determined. , To effectively determine the threshold, the frame is marked as a target candidate frame. Transfer to HSV space; otherwise, execute the computing power release mechanism and discard the frame directly.

[0013] Preferably, S3 includes: S31. A ROI extraction mechanism based on the HSV color space was constructed. First, the ROI extraction mechanism was... Converting to HSV space, a color mask function is constructed based on preset hydraulic instrument features. :

[0014] in For pixel coordinates, These are the hue, saturation, and brightness channel values, respectively. , The target color threshold range is defined. Then, morphological closing operations are performed on the mask image to fill internal holes, and the set of connected component contours is extracted. .

[0015] S32. To ensure that the extracted data from multiple dashboards is consistent with the visual reading logic of the physical panel, a system based on the bounding rectangle coordinates is constructed. Spatial clustering ranking model, defining the ranking weight index of contours. :

[0016] in, The threshold for row clustering tolerance. This is a column-level amplification weighting factor. The system follows... Arrange the images in ascending order and then cut out a structured sequence of ROI subgraphs.

[0017] S33. Introduce an area adaptive dynamic scaling algorithm, the first... The actual width and height of each effective ROI region are: The system presets the optimal pixel area threshold for large multimodal models to clearly read small characters as follows: .

[0018] Calculate the adaptive scaling factor for this ROI. :

[0019] in The function is designed to round up, ensuring that the multiple is sufficient to cover the minimum requirement. Ensure that large images are not abnormally reduced; based on the obtained A cubic interpolation algorithm was used to resample and enlarge the pixels in this region to obtain a high-resolution local feature map. .

[0020] Preferably, S4 includes: S41. Enlarged version The data are fed into the OCR engine and the large local visual model for parallel analysis. The large model is required to output the sluice gate's core parameters, such as status, opening value, left load, and right load, in JSON format. S42. In practical industrial applications, large models often cause incomplete JSON strings to be output due to token truncation, leading to program crashes; suppose the original output string of the large model is... The system constructs a set of reverse key-value pair matches based on regular expressions. :

[0021] in, This represents a collection of matching results, which stores all successfully matched key-value pairs; It is a regular expression iterative search function that scans the entire string and finds all segments that match the set rules; Represents the raw string output by the Large Model (LLM); These are the specific matching rules. and the following For raw string declarations in Python, ensure that special characters (such as backslashes) inside are not incorrectly escaped; This part is used to match and extract the "key" name, which is intended to match a continuous set of characters (usually field names composed of letters and Chinese characters, such as "status" or "encoding"). This is a connector used to match keys and values; This is used to match "value", and the intention is to match the specific content within the quotation marks.

[0022] like If not empty, extract the last complete match from the set. End position .

[0023] The system in Forced cut Discard any remaining incomplete or garbled characters; then define the set of necessary fields for water conservancy inspection: Calculate the set of fields that have been successfully parsed. The missing set is obtained. .right For each item in the string, the system automatically appends a preset safety default value to the end of the truncated string and forcibly appends a closing symbol \n}. This outputs a structured and complete set of repairable features. .

[0024] Preferably, S5 includes: S51. Extract the absolutely precise data from S4. Hydrological and hydraulic physical quantities such as opening degree and load are converted into natural language prompts.

[0025] S52. Construct a mandatory anti-hallucination multimodal cue word base:

[0026] in Represents text vector concatenation. Basic task instructions, This is an instruction that forces the model to use a given value and prohibits re-guessing.

[0027] S53. Will Compared with the original image The multimodal large model is input again. This time, the model is relieved of the burden of low-level digital recognition and focuses on global logical reasoning. Finally, it outputs accurate judgments on the overall safety status of water conservancy equipment, abnormal ambient temperature, and potential equipment hazards.

[0028] Preferably, S6 includes: S61. Extract the highest temperature sequence from all keyframes. Calculate the global maximum temperature With average temperature .

[0029] S62. Assemble temperature statistics, equipment type set, and the list of abnormal hazards generated in S5 into system-level snapshot data, input it into a large text language model with hundreds of billions of parameters for global diagnosis, and the LLM outputs a comprehensive judgment text containing defect analysis and handling opinions. .

[0030] S63. Call the typesetting engine framework, High-resolution partial ROI screenshot extracted The corresponding identification value matrix and global early warning extreme values ​​are automatically filled into the preset "Water Conservancy Hub Robot Dog Inspection Report" Word template according to the mapping anchor points, completing the fully automatic generation and export of the report.

[0031] Preferably, the method further includes: S7. Based on the water conservancy inspection report, the water conservancy monitoring experience is converted into structured knowledge slices, and then seamlessly updated to the RAG dynamic knowledge base through vectorized parsing.

[0032] Preferably, S7 includes: S71. Extract the security timestamp of this round of inspections. The diagnostic conclusions, extreme temperature values, and hazard status of S6 are merged into an unstructured long text. .

[0033] S72. Introduce a sliding window segmentation algorithm based on semantic boundary awareness. Perform document slicing; set the window size to [size missing]. Step size is It can be divided into multiple sub-text blocks:

[0034] S73. Call the embedding model to combine various The data is mapped to a high-dimensional feature vector and carries the set water conservancy document tags (such as ragflow_document_ids). It is dynamically added to the RAG (Retrieval Enhanced Generation) dynamic knowledge base through the API interface, providing a solid "prior memory" support for tracing the fault history of water conservancy equipment in the future.

[0035] The inspection and diagnosis system based on local feature enhancement and heterogeneous large models includes: The multi-source data receiving and preprocessing module is used to receive inspection image streams collected by multi-source terminals in real time, perform color space alignment and adaptive format preprocessing, and construct the image sequence to be analyzed. The panoramic target global coarse screening module is used to perform the first round of concurrent inference on the image sequence based on the multimodal visual large model, to perform global coarse screening and classification of device types, and to obtain the locked target; The adaptive extraction and feature enhancement module is used to establish an adaptive region of interest extraction mechanism based on HSV color space mask and morphological closing operation for locked targets. Through the bounding rectangle space clustering and sorting and threshold interpolation amplification algorithm, it separates and enhances high-definition local feature maps of small key components. The dual-engine recognition and JSON self-repair module is used to construct a large local visual model and a dual-engine OCR collaborative recognition mechanism to extract features in parallel from the magnified high-definition local feature map. It also establishes a dynamic regular truncation self-repair model for the non-standard JSON output of the large model, and obtains a repaired feature set through reverse key-value pair matching and automatic completion of missing fields. The prior injection and secondary anti-hallucination reasoning module is used to establish a prior feature injection mechanism, drive the large model to perform non-hallucination secondary joint reasoning based on the restorative feature set, and output an abnormal potential list. The global diagnosis and automatic report generation module is used to perform in-depth global logic chain diagnosis based on the temporal features and list of abnormal hazards of high-definition local feature maps, input into a large language model, and adaptively generate water conservancy inspection reports. The knowledge slicing and RAG knowledge base update module is used to convert water conservancy monitoring experience into structured knowledge slices based on water conservancy inspection reports, and then seamlessly update them to the RAG dynamic knowledge base through vectorized parsing.

[0036] As can be seen from the above technical solution, the present invention provides an inspection and diagnosis method based on local feature enhancement and heterogeneous large models. Compared with the prior art, the present invention has the following advantages: 1. This invention uses HSV color space masking and morphological algorithms to accurately locate and adaptively magnify tiny key areas in panoramic images, effectively solving the problem of visual illusions of small features in large models.

[0037] 2. This invention designs a dynamic regularization truncation and self-repair model for large-scale non-standard output, which ensures extremely high stability of industrial-grade structured feature data extraction and prevents the parsing pipeline from crashing.

[0038] 3. This invention establishes a collaborative mechanism between prior feature injection and multimodal large model, which perfectly decouples and deeply integrates the low-level precise numerical extraction with the high-order global logic judgment.

[0039] 4. This invention automatically generates graphic inspection reports through a fully automated typesetting engine, and realizes the self-growth and long-term time-series tracking of inspection experience through the RAG dynamic knowledge base. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the inspection and diagnosis method based on local feature enhancement and heterogeneous large models of the present invention.

[0041] Figure 2 This is a system block diagram of the inspection and diagnosis system based on local feature enhancement and heterogeneous large model of the present invention.

[0042] Figure 3 This is a visible light image of the control cabinet 1 in this invention.

[0043] Figure 4 This is a thermal imaging image of the control cabinet 1 in this invention.

[0044] Figure 5 This is a visible light image of the control cabinet 2 in this invention.

[0045] Figure 6 This is a thermal imaging image of the control cabinet 2 in this invention.

[0046] Figure 7 This is a visible light image of the fire protection facilities in this invention.

[0047] Figure 8 This is a thermal imaging image of the fire protection facilities in this invention.

[0048] Figure 9 This is a visible light image of the control cabinet 3 in this invention.

[0049] Figure 10 This is a thermal imaging image of the control cabinet 3 in this invention.

[0050] Figure 11 This is a pie chart illustrating the temperature risk statistics and analysis of this invention.

[0051] Figure 12 This is a pie chart showing the statistical analysis of appearance risks in this invention.

[0052] Figure 13 The scatter plot of the local feature space dimension and its kernel density estimation contour plot of the present invention. Figure 14 The image shows a multidimensional polar coordinate radar image before and after prior feature injection for this invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0054] like Figure 1 As shown, the inspection and diagnosis method based on local feature enhancement and heterogeneous large models in this embodiment includes the following steps: S1. Receive inspection image streams collected from multiple sources in real time, perform color space alignment and adaptive format preprocessing, and construct the image sequence to be analyzed; S2. Based on the multimodal visual large model, the image sequence is subjected to the first round of concurrent inference, and global coarse screening and classification of device types are performed to obtain the locked target; S3. For the locked target, an adaptive region of interest extraction mechanism based on HSV color space mask and morphological closing operation is established. Through the bounding rectangle space clustering sorting and threshold interpolation amplification algorithm, high-definition local feature maps of small key components are separated and enhanced. S4. Construct a local visual large model and OCR dual-engine collaborative recognition mechanism to extract features in parallel from the magnified high-definition local feature map, and establish a dynamic regular truncation self-repair model for the non-standard JSON output of the large model. Through reverse key-value pair matching and automatic completion of missing fields, a repaired feature set is obtained. S5. Establish a prior feature injection mechanism to drive the large model to perform hallucination-free secondary joint reasoning based on the repair feature set and output an abnormal potential list; S6. Based on the temporal features and list of abnormal hazards in the high-definition local feature map, input the data into the large language model to perform a global logic chain in-depth diagnosis and adaptively generate a water conservancy inspection report.

[0055] Specifically, S1 includes: S11. Establish an asynchronous image stream monitoring mechanism for the complex network environment of water conservancy hubs, to receive visible light and infrared thermal imaging dual-light data transmitted back by the robot dog in real time, and to receive the original image sequence transmitted back by the inspection robot. ; Detecting the tensor dimension of an image If color channel dimensions Establish a color space mapping matrix Force mapping of single-channel grayscale or four-channel RGBA to a standard BGR format matrix :

[0056] S12. Construct an image tensor buffer and assign it to the image based on the timestamp. The data is serialized and encapsulated, stored in the queue to be analyzed, and then handed over to the downstream asynchronous thread pool for processing.

[0057] S21. Let the first... The features of a frame panoramic image are The basic category suggestion keywords are The probability of the target device's existence is calculated using a multimodal large model:

[0058] S22. Establish a set of target category identifiers If the set of device type categories parsed by the large model is consistent with... The intersection of the two sets of elements is not empty, which means that the condition is determined. , To effectively determine the threshold, the frame is marked as a target candidate frame. Switch to S3; otherwise, execute the computing power release mechanism and discard the frame directly.

[0059] Specifically, S3 includes: S31. A ROI extraction mechanism based on the HSV color space was constructed. First, the ROI extraction mechanism was... Converting to HSV space, a color mask function is constructed based on preset hydraulic instrument features. :

[0060] in For pixel coordinates, These are the hue, saturation, and brightness channel values, respectively. , The target color threshold range is defined. Then, morphological closing operations are performed on the mask image to fill internal holes, and the set of connected component contours is extracted. .

[0061] S32. To ensure that the extracted data from multiple dashboards is consistent with the visual reading logic of the physical panel, a system based on the bounding rectangle coordinates is constructed. Spatial clustering ranking model, defining the ranking weight index of contours. :

[0062] in, The threshold for row clustering tolerance. This is a column-level amplification weighting factor. The system follows... Arrange the images in ascending order and then cut out a structured sequence of ROI subgraphs.

[0063] S33. Introduce an area adaptive dynamic scaling algorithm, the first... The actual width and height of each effective ROI region are: The system presets the optimal pixel area threshold for large multimodal models to clearly read small characters as follows: .

[0064] Calculate the adaptive scaling factor for this ROI. :

[0065] in The function is designed to round up, ensuring that the multiple is sufficient to cover the minimum requirement. Ensure that large images are not abnormally reduced; based on the obtained A cubic interpolation algorithm was used to resample and enlarge the pixels in this region to obtain a high-resolution local feature map. .

[0066] like Figure 13As shown, this invention provides a visual analysis of the distribution of extracted local features in two-dimensional space; Figure 13 It shows the scatter plot distribution and kernel density estimation contour plot of the local feature space dimension.

[0067] Experimental results show that the extracted key micro-component features exhibit a significant clustering effect in space. Green scatter points represent dashboard ROI clusters, and purple scatter points represent screen status ROI clusters. The size of the scatter points maps to the relative area of ​​the bounding rectangular mask of the ROI. From the bottom-level KDE red contour distribution, two independent probability peaks with extremely high density can be clearly observed, and there is a clear decision boundary between the two clusters, with no severe feature aliasing.

[0068] This result fully demonstrates that the adaptive extraction mechanism constructed in this invention can effectively overcome the interference of complex background environments such as reflections and water stains in water conservancy projects, and possesses excellent intra-class aggregation and inter-class separation. This step successfully isolates discrete and small local targets from the panoramic image, significantly improving the signal-to-noise ratio and laying a high-quality data foundation for subsequent high-precision parallel feature extraction using dual engines.

[0069] Specifically, S4 includes: S41. Enlarged version The data are fed into the OCR engine and the local visual large model for parallel analysis. The large model is required to output the sluice gate's core parameters, such as status, opening value, left load, and right load, in JSON format. S42. In practical industrial applications, large models often cause incomplete JSON strings to be output due to token truncation, leading to program crashes; suppose the original output string of the large model is... The system constructs a set of reverse key-value pair matches based on regular expressions. :

[0070] like If not empty, extract the last complete match from the set. End position .

[0071] The system in Forced cut Discard any remaining incomplete or garbled characters; then define the set of necessary fields for water conservancy inspection: ; Calculate the set of fields that have been successfully parsed The missing set is obtained. .right For each item in the string, the system automatically appends a preset safety default value to the end of the truncated string and forcibly appends a closing symbol \n}. This outputs a structured and complete set of repairable features. .

[0072] Specifically, S5 includes: S51. Extract the absolutely precise data from S4. Hydrological and hydraulic physical quantities such as opening degree and load are converted into natural language prompts.

[0073] S52. Construct a mandatory anti-hallucination multimodal cue word base:

[0074] in Represents text vector concatenation. Basic task instructions, This is an instruction that forces the model to use a given value and prohibits re-guessing.

[0075] S53. Will Compared with the original image The multimodal large model is input again. This time, the model is relieved of the burden of low-level digital recognition and focuses on global logical reasoning. Finally, it outputs accurate judgments on the overall safety status of water conservancy equipment, abnormal ambient temperature, and potential equipment hazards.

[0076] like Figure 14 As shown, multimodal large language models often produce hallucination phenomena when processing unstructured images of complex industrial scenes due to the lack of local physical fact constraints. Figure 14 Using a multi-dimensional polar radar chart, the performance differences between direct inference using a single first-round large model (gray dashed line) and the proposed structured prior Prompt-based secondary fusion inference (green solid line) in five core dimensions were quantitatively compared.

[0077] Observing the radar chart reveals that the traditional direct reasoning method has significant shortcomings in terms of hallucination suppression rate and numerical logic rationality. This is because large models tend to overlook small, crucial local information when faced with a global high-dimensional image, leading to generated results that deviate from physical reality. However, after introducing the secondary fusion reasoning mechanism, all indicators have been substantially improved, and the area of ​​the formed green polygon significantly covers and surrounds the baseline polygon; in particular, the hallucination suppression rate and the consistency of state determination approach extremely high levels.

[0078] From a mechanistic perspective, the precisely extracted local instrument values ​​and screen states are transformed into structured prior information and injected into the large model as contextual constraints. This mechanism essentially sets strict physical knowledge anchors for the large model's generation space, effectively limiting its divergent output and significantly reducing the probability of generating false content. Experimental data show that this multimodal contextual fusion strategy can significantly enhance the robustness and logical rigor of the diagnostic model, meeting the high reliability engineering requirements of water conservancy project inspections.

[0079] Specifically, S6 includes: S61. Extract the highest temperature sequence from all keyframes. Calculate the global maximum temperature With average temperature .

[0080] S62. Assemble the temperature statistics, equipment type set, and the list of abnormal hazards generated in S5 into system-level snapshot data, and input it into a large text language model with hundreds of billions of parameters for global diagnosis. If the parameters are valid, the LLM outputs a comprehensive judgment text containing defect analysis and handling opinions. Otherwise, return to continue locking the image.

[0081] S63. Call the typesetting engine framework, High-resolution partial ROI screenshot extracted The corresponding identification value matrix and global early warning extreme values ​​are automatically filled into the preset "Water Conservancy Hub Robot Dog Inspection Report" Word template according to the mapping anchor points, completing the fully automatic generation and export of the report.

[0082] Specifically, the methods also include: S7. Based on the water conservancy inspection report, the water conservancy monitoring experience is converted into structured knowledge slices, and then seamlessly updated to the RAG dynamic knowledge base through vectorized parsing.

[0083] Specifically, S7 includes: S71. Extract the security timestamp of this round of inspections. The diagnostic conclusions, extreme temperature values, and hazard status of S6 are merged into an unstructured long text. .

[0084] S72. Introduce a sliding window segmentation algorithm based on semantic boundary awareness. Perform document slicing. Assume the window size is [size missing]. Step size is It can be divided into multiple sub-text blocks:

[0085] S73. Call the embedding model to combine various The data is mapped to a high-dimensional feature vector and carries the set water conservancy document tags (such as ragflow_document_ids). It is dynamically added to the RAG (Retrieval Enhanced Generation) dynamic knowledge base through the API interface, providing a solid "prior memory" support for tracing the fault history of water conservancy equipment in the future.

[0086] like Figure 2 As shown, the inspection and diagnosis system based on local feature enhancement and heterogeneous large model includes: The multi-source data receiving and preprocessing module is used to receive inspection image streams collected by multi-source terminals in real time, perform color space alignment and adaptive format preprocessing, and construct the image sequence to be analyzed. The panoramic target global coarse screening module is used to perform the first round of concurrent inference on the image sequence based on the multimodal visual large model, to perform global coarse screening and classification of device types, and to obtain the locked target; The adaptive extraction and feature enhancement module is used to establish an adaptive region of interest extraction mechanism based on HSV color space mask and morphological closing operation for locked targets. Through the bounding rectangle space clustering and sorting and threshold interpolation amplification algorithm, it separates and enhances high-definition local feature maps of small key components. The dual-engine recognition and JSON self-repair module is used to construct a large local visual model and a dual-engine OCR collaborative recognition mechanism to extract features in parallel from the magnified high-definition local feature map. It also establishes a dynamic regular truncation self-repair model for the non-standard JSON output of the large model, and obtains a repaired feature set through reverse key-value pair matching and automatic completion of missing fields. The prior injection and secondary anti-hallucination reasoning module is used to establish a prior feature injection mechanism, drive the large model to perform non-hallucination secondary joint reasoning based on the restorative feature set, and output an abnormal potential list. The global diagnosis and automatic report generation module is used to perform in-depth global logic chain diagnosis based on the temporal features and list of abnormal hazards of high-definition local feature maps, input into a large language model, and adaptively generate water conservancy inspection reports. The knowledge slicing and RAG knowledge base update module is used to convert water conservancy monitoring experience into structured knowledge slices based on water conservancy inspection reports, and then seamlessly update them to the RAG dynamic knowledge base through vectorized parsing.

[0087] In practical applications, taking a reservoir in Anhui Province as an example, the following inspection is conducted: like Figure 3 and Figure 4 As shown, based on the visible light image and thermal imaging image of control cabinet 1, it can be seen that: Safety alert: Normal; 1. Equipment inspection: The control cabinet door is tightly closed, the panel indicator lights and display screen are normal, the buttons and knobs are intact, the markings are clear, and there is no looseness, burning or overheating discoloration. The inspection results are normal. 2. Temperature check: The highest temperature is 20.9℃, and the lowest temperature is 17.3℃. The alarm level is normal, and the temperature status is normal. 3. Check the opening value of the LCD screen: (1) Status: lower limit, opening value: 000.02, left load: 000.0, right load: 000.0; (2) Status: lower limit, opening value: 000.00, left load: 000.0, right load: -000.0, the display and opening value status are normal.

[0088] like Figure 5 and Figure 6 As shown, based on the visible light image and thermal imaging image of control cabinet 2, it can be seen that: Safety reminder: Be aware of high temperature; alarm level is "caution". 1. Equipment inspection: The control cabinet is in good condition, the three LCD screens are functioning normally, the indicator lights, buttons and knobs are intact, there is no rust, damage or looseness, the operation is normal, and the inspection results are normal. 2. Temperature check: The highest temperature is 34.8℃, and the lowest temperature is 17.4℃. Alarm level: Note. Temperature status is normal. 3. Check the opening value of the LCD screen: (1) Status: lower limit, opening value: 000.02, left load: 000.0, right load: 000.0; (2) Status: lower limit, opening value: 000.00, left load: 000.0, right load: -000.0; (3) Status: lower limit, opening value: -00.00, left load: 003.1, right load: -001.5, the display and opening value status are normal.

[0089] like Figure 7 and Figure 8 As shown, based on the visible light and thermal images of the fire protection facilities, it can be concluded that: Safety reminder: Be aware of high temperature; alarm level is "caution". 1. Equipment inspection: The fire extinguisher box is intact, the markings are clear, there are no obstructions around it, and there is no rust, damage or obstruction. The fire protection facilities are in normal condition, and the inspection results are normal. 2. Temperature check: The highest temperature is 34.6℃, and the lowest temperature is 21.2℃. Alarm level: Note. Temperature status is normal.

[0090] like Figure 9 and Figure 10 As shown, based on the visible light image and thermal imaging image of control cabinet 3, it can be seen that: Safety reminder: Be aware of high temperature; alarm level is "caution". 1. Equipment Inspection: The control cabinet door is tightly closed; all three LCD screens display the correct information. Normally, the buttons and knobs are intact, the indicator lights are in normal condition, and there are no signs of rust, damage, or burning. The inspection results are normal. 2. Temperature check: The highest temperature is 37.4℃, and the lowest temperature is 20.8℃. Alarm level: Note. Temperature status is normal. 3. Check the opening value of the LCD screen: (1) Status: lower limit, opening value: 000.02, left load: 000.0, right load: -000.0; (2) Status: lower limit, opening value: 000.00, left load: 000.0, right load: -000.0; (3) Status: lower limit, opening value: -00.00, left load: 002.6, right load: -003.4, the display and opening value status are normal.

[0091] In summary, as Figure 11 As shown in the temperature risk statistics and analysis chart, the generated list of temperature hazard risks is as follows: Risk level Judgment criteria quantity percentage Involving inspection targets Recommendations for handling normal Maximum temperature ≤30℃ 1 25% Control Cabinet 1 Continue to monitor and conduct inspections as planned. Pay more attention 30℃ < maximum temperature ≤ 40℃ 3 75% Control cabinet 2, control cabinet 3, fire protection equipment 1 Including it in subsequent review, and monitoring the temperature rise trend. Risk exists Maximum temperature > 40℃ 0 0% - Arrange on-site retesting to check the load, wiring, and heat dissipation. Table 1. List of Potential Risks of Abnormal Temperature in Hydraulic Equipment like Figure 12 As shown in the temperature risk statistics and analysis chart, the generated list of potential appearance hazards is as follows: Appearance Classification Judgment criteria quantity percentage Involving inspection targets Recommendations for handling normal No potential hazards such as damage, rust, obstruction, or looseness were found. 4 100% Control cabinet 1, control cabinet 2, control cabinet 3, fire protection equipment 1 Maintain the current inspection frequency Needs attention Items indicating minor potential issues or requiring manual review 0 0% - Arrange for manual review and track changes. Abnormal risks Identified as an abnormal state or with a clear security risk 0 0% - Timely rectification, closed-loop management after re-inspection and confirmation. Table 2. List of Potential Risks of Abnormal Temperature in Hydraulic Equipment In summary, the status of the LCD display on control cabinet 1 needs to be verified. Problem description: The control cabinet LCD screen displays the opening value as the lower limit. It is necessary to confirm whether the feedback value is consistent with the actual opening value based on the actual on-site working conditions.

[0092] Recommended action: Manually verify valve opening or mechanical position on-site to rule out sensor drift or signal transmission faults, and ensure accurate control system indications.

[0093] 2. The temperature of fire-fighting equipment 1 needs to be continuously monitored. Problem description: The highest temperature recorded by thermal imaging of the control cabinet is 34.8 degrees Celsius, which is within the allowable range, but continuous monitoring based on operating conditions is still necessary to prevent abnormal temperature rise.

[0094] Recommendations: Record operating temperature data regularly and compare it with historical peak values. If a continuous rise occurs, promptly investigate potential causes such as overload or poor contact.

[0095] 3. The temperature of control cabinet 2 needs close monitoring. Problem description: The highest temperature of the control cabinet reached 37.4 degrees Celsius, which is higher than that of other equipment. We need to pay attention to whether there are any safety hazards in the subsequent temperature rise trend.

[0096] Recommended solutions: Check the internal load current and the tightness of the wiring terminals, ensure good ventilation and heat dissipation, and focus on monitoring the temperature changes of the equipment during subsequent inspections.

[0097] 4. The appearance of control cabinet 3 needs to be checked regularly. Problem description: The fire protection facilities are in good condition with no rust or damage, but it is necessary to ensure that there are no debris piled up inside the enclosure that would affect emergency access and the integrity of the equipment.

[0098] Recommendations: Clean up obstructions around the enclosure monthly, check the clarity and expiration of signage, and ensure that fire safety facilities are always in a usable and compliant condition.

[0099] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk).

[0100] It should be noted that in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0101] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0102] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for inspection and diagnosis based on local feature enhancement and heterogeneous large models, characterized in that, Includes the following steps: S1. Receive inspection image streams collected from multiple sources in real time, perform color space alignment and adaptive format preprocessing, and construct the image sequence to be analyzed; S2. Based on the multimodal visual large model, the image sequence is subjected to the first round of concurrent inference, and global coarse screening and classification of device types are performed to obtain the locked target; S3. Based on the locked target, establish an adaptive region of interest extraction mechanism based on HSV color space mask and morphological closing operation. Through the bounding rectangle space clustering sorting and threshold interpolation amplification algorithm, separate and enhance the high-definition local feature map of small key components. S4. Construct a local visual large model and OCR dual-engine collaborative recognition mechanism to extract features in parallel from the magnified high-definition local feature map, and establish a dynamic regular truncation self-repair model for the non-standard JSON output of the large model. Through reverse key-value pair matching and automatic completion of missing fields, a repaired feature set is obtained. S5. Establish a prior feature injection mechanism to drive the large model to perform hallucination-free secondary joint reasoning based on the repair feature set and output an abnormal potential list; S6. Based on the temporal features and list of abnormal hazards in the high-definition local feature map, input the data into the large language model to perform a global logic chain in-depth diagnosis and adaptively generate a water conservancy inspection report.

2. The inspection and diagnosis method based on local feature enhancement and heterogeneous large model according to claim 1, characterized in that: S1 includes: S11. Establish an asynchronous image stream monitoring mechanism for the complex network environment of water conservancy hubs, to receive visible light and infrared thermal imaging dual-light data transmitted back by the robot dog in real time, and to receive the original image sequence transmitted back by the inspection robot. ; Detecting the tensor dimension of an image If color channel dimensions Establish a color space mapping matrix Force mapping of single-channel grayscale or four-channel RGBA to a standard BGR format matrix : S12. Construct an image tensor buffer and assign it to the image based on the timestamp. The data is serialized and encapsulated, stored in the queue to be analyzed, and then handed over to the downstream asynchronous thread pool for processing.

3. The inspection and diagnosis method based on local feature enhancement and heterogeneous large model according to claim 2, characterized in that: S2 includes: S21. Let the first... The features of a frame panoramic image are The basic category suggestion keywords are The probability of the target device's existence is calculated using a multimodal large model: ； S22. Establish a set of target category identifiers: If the set of device type categories parsed from the large model is consistent with... The intersection of the two sets of elements is not empty, which means that the condition is determined. , To effectively determine the threshold, the frame is marked as a target candidate frame. Transfer to HSV space; otherwise, execute the computing power release mechanism and discard the frame directly.

4. The inspection and diagnosis method based on local feature enhancement and heterogeneous large model according to claim 3, characterized in that: S3 includes: S31. A ROI extraction mechanism based on the HSV color space was constructed. First, the ROI extraction mechanism was... Converting to HSV space, a color mask function is constructed based on preset hydraulic instrument features. : in For pixel coordinates, These are the hue, saturation, and brightness channel values, respectively. , The target color threshold range; Subsequently, morphological closing operations are performed on the mask image to fill the internal holes, and the set of connected component contours is extracted. ; S32. Constructing coordinates based on the circumscribed rectangle Spatial clustering ranking model, defining the ranking weight index of contours. : in, The threshold for row clustering tolerance. This is a column-level amplification weighting factor; The system follows Arrange in ascending order and crop out the structured ROI sub-image sequence in sequence; S33. Introduce an area adaptive dynamic scaling algorithm, the first... The actual width and height of each effective ROI region are: The system presets the optimal pixel area threshold for large multimodal models to clearly read small characters as follows: ; Calculate the adaptive scaling factor of the ROI. : in, It is a rounding function. This indicates that large images will not be abnormally reduced; Based on the obtained A cubic interpolation algorithm is used to resample and enlarge the pixels of the region to obtain a high-resolution local feature map. .

5. The inspection and diagnosis method based on local feature enhancement and heterogeneous large model according to claim 4, characterized in that: S4 includes: S41. Enlarged version The data are fed into the OCR engine and the local visual large model for parallel analysis, and the large model outputs the core parameters of the sluice gate. S42. Let the original output string of the large model be... Construct a set of reverse key-value pair matches based on regular expressions. : in, Represents the set of matching results. It is an iterative search function based on regular expressions. This represents the raw string output by the Large Model (LLM). It refers to the specific matching rules, if If not empty, extract the last complete match from the set. End position ; exist Forced cut Discard any incomplete or garbled characters; define the set of necessary fields for water conservancy inspection. ; Calculate the set of fields that have been successfully parsed The missing set is obtained. ;right For each item in the truncated string, the system automatically appends a preset safety default value to the end, thereby outputting a structured and complete repair feature set. .

6. The inspection and diagnosis method based on local feature enhancement and heterogeneous large model according to claim 5, characterized in that: S5 includes: S51. Extract the absolutely precise data from S4. Hydrological and hydraulic physical quantities are converted into natural language prompts; S52. Construct a mandatory anti-hallucination multimodal cue word base: in, Represents text vector concatenation. Basic task instructions, This is an instruction to force the model to use given values ​​and prevent re-guessing; S53. Will Compared with the original image Input the multimodal large model again, and finally output a list of abnormal hidden dangers of water conservancy equipment.

7. The inspection and diagnosis method based on local feature enhancement and heterogeneous large model according to claim 6, characterized in that: S6 includes: S61. Extract the highest temperature sequence from all keyframes. Calculate the global maximum temperature With average temperature ; S62. Assemble temperature statistics, equipment type sets, and lists of potential anomalies into system-level snapshot data, input it into a large text language model for global diagnosis, and output a comprehensive judgment text containing defect analysis and handling suggestions. ; S63. Call the typesetting engine framework, High-resolution partial ROI screenshot extracted The corresponding identification value matrix and global early warning extreme values ​​are automatically filled into the preset water conservancy hub robot dog inspection report template according to the mapping anchor points, completing the fully automatic generation and export of the report.

8. The inspection and diagnosis method based on local feature enhancement and heterogeneous large model according to claim 7, characterized in that: The method also includes: S7. Based on the water conservancy inspection report, the water conservancy monitoring experience is converted into structured knowledge slices, and then seamlessly updated to the RAG dynamic knowledge base through vectorized parsing.

9. The inspection and diagnosis method based on local feature enhancement and heterogeneous large model according to claim 8, characterized in that: S7 includes: S71. Extract the security timestamp of this round of inspections. The inspection report, extreme temperature values, and hazard status will be merged into an unstructured long text. ; S72. Introduce a sliding window segmentation algorithm based on semantic boundary awareness. Perform document slicing, and set the window size to [size missing]. Step size is It can be divided into multiple sub-text blocks: S73. Call the embedding model to combine various The mapping is converted into a high-dimensional feature vector, carrying the set water conservancy document tags, and dynamically appended to the RAG dynamic through the API interface.

10. A patrol diagnosis system based on local feature enhancement and heterogeneous large model, employing the patrol diagnosis method based on local feature enhancement and heterogeneous large model as described in any one of claims 1-9, characterized in that, include: The multi-source data receiving and preprocessing module is used to receive inspection image streams collected by multi-source terminals in real time, perform color space alignment and adaptive format preprocessing, and construct the image sequence to be analyzed. The panoramic target global coarse screening module is used to perform the first round of concurrent inference on the image sequence based on the multimodal visual large model, to perform global coarse screening and classification of device types, and to obtain the locked target; The adaptive extraction and feature enhancement module is used to establish an adaptive region of interest extraction mechanism based on HSV color space mask and morphological closing operation for locked targets. Through the bounding rectangle space clustering and sorting and threshold interpolation amplification algorithm, it separates and enhances high-definition local feature maps of small key components. The dual-engine recognition and JSON self-repair module is used to construct a large local visual model and a dual-engine OCR collaborative recognition mechanism to extract features in parallel from the magnified high-definition local feature map. It also establishes a dynamic regular truncation self-repair model for the non-standard JSON output of the large model, and obtains a repaired feature set through reverse key-value pair matching and automatic completion of missing fields. The prior injection and secondary anti-hallucination reasoning module is used to establish a prior feature injection mechanism, drive the large model to perform non-hallucination secondary joint reasoning based on the restorative feature set, and output an abnormal potential list. The global diagnosis and automatic report generation module is used to perform in-depth global logic chain diagnosis based on the temporal features and list of abnormal hazards of high-definition local feature maps, input into a large language model, and adaptively generate water conservancy inspection reports. The knowledge slicing and RAG knowledge base update module is used to convert water conservancy monitoring experience into structured knowledge slices based on water conservancy inspection reports, and then seamlessly update them to the RAG dynamic knowledge base through vectorized parsing.

Images