Intelligent inspection system and method for water affair facilities based on multi-modal interaction

Abstract

The invention discloses a water affair facility intelligent inspection system and method based on multi-modal interaction, and relates to the field of water affair facility management, and the system comprises a collection module which is used for collecting appearance state data and operation parameter data of a water affair facility, and storing the collected data; the regulation and control module is used for receiving an operation instruction of an inspector, regulating and controlling the collection range and precision of the collection module in real time based on the operation instruction, and synchronously carrying out classification and redundancy elimination processing on the two types of collected data; the appearance and operation data of the water affair facility are collected in a non-contact mode through a handheld device or an unmanned aerial vehicle of an inspector, the collection focal length, frequency and flight path are adjusted in real time to guarantee the data to be comprehensive and accurate, the collected data are classified to remove redundancy and improve the quality, and the facility anomaly degree is calculated in combination with historical normal data. And abnormal features are accurately identified and suspected abnormal types are determined.

Description

Technical Field

[0001] This invention relates to the field of water facility management technology, specifically to an intelligent inspection system and method for water facilities based on multimodal interaction. Background Technology

[0002] With the accelerated development of smart cities, the water industry, as a core component of urban infrastructure, is facing a critical stage of digital transformation. Traditional water engineering suffers from problems such as data silos, low operational efficiency, and delayed emergency response, urgently requiring the use of intelligent technologies to achieve business collaboration and decision optimization. Currently, water systems involve complex scenarios such as water source management, pipeline monitoring, sewage treatment, and flood and drought control. The integration and efficient utilization of multi-source heterogeneous data (such as sensor data, geographic information, video surveillance, and user feedback) has become a pain point for the industry.

[0003] However, existing water management platforms generally suffer from problems such as limited interaction methods (mainly charts), insufficient cross-departmental collaboration capabilities, and low levels of 3D visualization, making it difficult to meet the needs of real-time dynamic management and emergency command.

[0004] To address this, we propose an intelligent inspection system and method for water facilities based on multimodal interaction. Summary of the Invention

[0005] In view of the above-mentioned shortcomings of the existing technology, the present invention provides an intelligent inspection system and method for water facilities based on multimodal interaction, which can effectively solve the problems of the existing technology.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions; This invention discloses an intelligent inspection system for water facilities based on multimodal interaction, comprising: The system comprises the following modules: a data acquisition module for collecting and storing data on the appearance and operating parameters of water facilities; a control module for receiving operational instructions from inspection personnel, adjusting the acquisition range and accuracy of the acquisition module in real time based on these instructions, and simultaneously classifying and deduplicating the two types of data; an analysis module for receiving the classified and deduplicated data from the control module, identifying abnormal characteristics of water facilities, and determining suspected anomalies; a coordination module for creating a multimodal information interaction channel between inspection personnel and remote technicians, transmitting data collected by the acquisition module and processed by the control module, as well as analysis results from the analysis module; a recording module for encrypting and storing the acquired data, operational instructions, and anomaly analysis results, generating a unique inspection record identifier, and associating the inspection time and personnel information of the corresponding facility; and an optimization module for generating and setting the optimal path for the next inspection based on historical inspection data stored in the recording module, combined with the distribution location and operating status of the water facilities. Among them, the appearance status data and operating parameter data of water facilities are collected in a non-contact manner by inspection personnel or drones equipped with corresponding data collection equipment; The acquisition module is interactively connected to the control module via a wireless network. The control module is interactively connected to the analysis module via a wireless network. The analysis module is interactively connected to the coordination module via a wireless network. The coordination module is interactively connected to the recording module and the optimization module via a wireless network. The recording module is interactively connected to the optimization module via a wireless network.

[0007] Furthermore, during the non-contact acquisition phase, when the inspection personnel collect data using a handheld acquisition device, the acquisition module automatically adjusts the focal length and sampling frequency of the acquisition device according to the hand gesture instructions of the inspection personnel. When the inspection personnel select a specific area of ​​the pipeline using hand gesture instructions, the sampling frequency of the acquisition device automatically increases to a preset high-frequency range, and the resolution of the appearance image acquisition is synchronously matched with the size of the selected area. When the drone is equipped with data acquisition equipment, the data acquisition module first obtains the distribution coordinates of the water facilities to be inspected, automatically generates a pre-planned flight path covering all target facilities, and dynamically adjusts the flight altitude and shooting angle in real time according to the airspace environment during the flight, so that the appearance images of each water facility cover at least three different perspectives. At the same time, the continuously acquired appearance images are stitched together in real time to form a panoramic appearance image of the overall appearance of the facility and the details of key components.

[0008] Furthermore, the control module performs data classification and redundancy removal processes, following the following rules: The collected data is divided into primary categories based on appearance status data and operating parameter data. Then, the appearance status data is divided into secondary categories based on images of the main structure of the facility, images of auxiliary components, and environmental images. The operating parameter data is divided into secondary categories based on electrical parameters, mechanical parameters, metering parameters, and signal parameters. For data within the same secondary category, calculate the similarity between any two sets of data. When the similarity exceeds a preset similarity threshold, retain the data whose collection time is closer to the current inspection time and delete redundant data. The formula for calculating data similarity is: ; In the formula: These are the weighting coefficients. ∈ (0, 1); The feature similarity between the two sets of appearance status data is determined by calculating the matching degree of SIFT feature points extracted from the images; The numerical similarity between the two sets of operating parameter data; in, , This indicates the number of operating parameters under this secondary category. Let represent the values ​​of the i-th running parameter in each of the two sets of data. This represents the range of the normal value range of the i-th operating parameter.

[0009] Furthermore, the analysis module identifies abnormal features of water facilities to determine suspected anomaly types as follows: Feature extraction is performed on the appearance status data processed by the control module to obtain appearance feature parameters including at least the proportion of damaged area on the facility surface, the proportion of pixels in color abnormal areas, and the degree of component deformation. The processed operating parameter data is then standardized to obtain standardized electrical parameters, mechanical parameters, metering parameters, and signal parameters. The appearance feature parameters are combined with the standardized operating parameters to form the feature vector of the water facility. Where m is the total number of feature parameters, This is the quantized value of the first feature parameter; Obtain historical normal inspection data for a specified time interval. Based on this data, calculate the normal mean and standard deviation for each characteristic parameter. Then, determine the facility anomaly level based on the normal mean and standard deviation. , This represents the weight coefficient of the k-th feature parameter. This represents the quantized value of the k-th feature parameter. This represents the normal mean and normal standard deviation. Used to represent the deviation of the k-th feature parameter; when When the deviation exceeds the preset abnormal threshold, the facility is determined to have a suspected abnormality. At the same time, the three feature parameters with the largest deviations are selected according to the deviation of each feature parameter. Then, the suspected abnormality type of the water facility is determined according to the preset feature-abnormality mapping relationship.

[0010] Furthermore, the multimodal information interaction channels created by the coordination module include a voice interaction channel, an image sharing channel, and a parameter interaction channel; When the analysis module determines that the facility has a suspected anomaly and the anomaly level exceeds the preset high-risk threshold, the coordination module automatically increases the transmission frame rate of the image sharing channel to the preset high-frequency range, switches the voice interaction channel to low-latency mode with the latency controlled within the preset threshold, and suspends the update of non-related parameters in the parameter interaction channel. When a remote technician initiates an interactive command through any channel, the other two channels automatically synchronize the timestamp of the command with the associated object.

[0011] Furthermore, the recording module performs encrypted storage and unique identifier generation in accordance with the following: Symmetric encryption algorithms are used to encrypt the collected data, operation instructions, and anomaly analysis results. The encryption key is generated by combining the pre-set unique identification key of the inspector and the facility-specific key. The inspector identification key is bound to the inspector's fingerprint or face, and the facility-specific key is a pre-set unique key for each water facility. The encrypted data is stored in a distributed database, and the access permissions of the database nodes correspond one-to-one with the job permissions of the inspectors. After the collected data, operation instructions, and anomaly analysis results are encrypted and stored in the distributed database, the corresponding cached data in the source module of the collected data, operation instructions, and anomaly analysis results are all deleted. The unique inspection record identifier follows the following rules when generated: ,in A unique code indicating a water facility. Indicates the timestamp of the inspection operation. This indicates the unique employee number of the patrol inspector. It represents the unique inspection record identifier generated by the SHA-256 hash function, which is then associated with the inspection time, inspection personnel information, and corresponding facility code for storage.

[0012] Furthermore, the process by which the optimization module generates the optimal path for the next inspection is as follows: Based on the historical inspection data stored in the recording module, an inspection weight is assigned to each water facility. The weight is calculated as the product of the number of suspected anomalies in history, the functional importance of the facility, and the time since the last inspection. If any one or more of these values ​​are zero, then the zero value is not included in the product calculation. Synchronously set path constraints: the inspection path should not repeatedly pass through the same area; Finally, the deployment locations of each water facility are used as path nodes. Based on the inspection weight of each water facility, the corresponding path nodes are sorted from high to low, and the path nodes are connected in order according to the sorting results, and the connection results meet the set path constraints. When the connecting path between any two path nodes is not unique, the shorter path is selected first, and the remaining paths are used to satisfy the path constraints.

[0013] On the other hand, a smart inspection method for water facilities based on multimodal interaction includes: The system employs a non-contact approach, utilizing handheld or drone-mounted data collection devices by inspectors, to collect data on the appearance and operational parameters of water facilities. This data is initially stored. The system receives operational instructions from inspectors to adjust the collection module's range and accuracy. The collected data is categorized into primary categories based on appearance and operational parameters, further subdivided into secondary categories. Redundant data is then removed based on a data similarity calculation formula. Appearance feature parameters and standardized operational parameters are extracted from the processed data to form feature vectors. Facility anomaly is calculated; if the anomaly exceeds a preset threshold, the suspected anomaly type is determined based on the deviation of the feature parameters. A multimodal interactive channel (voice, image, parameters) is created to transmit data and analysis results. For high-risk anomalies, the channel transmission mode is optimized, and the timestamps and associated objects of interactive instructions from each channel are synchronized. Data is symmetrically encrypted using a key generated from a combination of the inspector's identity key and the facility's exclusive key and stored in a distributed database. A unique inspection record identifier containing the facility code, timestamp, and employee number is generated and associated with this identifier. Facility inspection weights are calculated based on the number of historical suspected anomalies, the facility's functional importance, and the time since the last inspection. Path nodes are sorted by weight, and the optimal inspection path for the next inspection is generated based on constraints.

[0014] Compared with the known prior art, the technical solution provided by this invention has the following beneficial effects: This invention provides an intelligent inspection system and method for water facilities based on multimodal interaction. During execution, the system and method collect the appearance and operational data of water facilities in a non-contact manner using handheld devices or drones by inspection personnel. The system adjusts the collection focus, frequency, and flight path in real time to ensure comprehensive and accurate data. The collected data is classified and redundant-free to improve quality. The system calculates the facility anomaly degree by combining historical normal data, accurately identifies abnormal features and determines suspected anomaly types, establishes a multimodal information interaction channel, optimizes the transmission mode to improve the collaboration efficiency between inspection and remote technical personnel in the event of high-risk anomalies, encrypts and stores the data and generates a unique inspection identifier to ensure security and traceability, and plans the optimal path based on historical inspection data and facility weights to avoid duplication and prioritize coverage of important facilities. Overall, the system improves the intelligence level, operational efficiency, and safety assurance capabilities of water facility inspection. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0016] Figure 1 This is a schematic diagram of the structure of an intelligent inspection system for water facilities based on multimodal interaction; Figure 2 This is a flowchart illustrating an intelligent inspection method for water facilities based on multimodal interaction. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0018] The present invention will be further described below with reference to embodiments.

[0019] Example 1: This embodiment presents an intelligent water facility inspection system based on multimodal interaction, such as... Figure 1 As shown, it includes: The data acquisition module is used to collect data on the appearance and operating parameters of water facilities and to store the collected data. During the non-contact acquisition phase, when the inspection personnel collect data using a handheld acquisition device, the acquisition module automatically adjusts the focal length and sampling frequency of the acquisition device according to the hand gesture instructions of the inspection personnel. When the inspection personnel select a specific area of ​​the pipeline using hand gesture instructions, the sampling frequency of the acquisition device automatically increases to the preset high frequency range, and the resolution of the appearance image acquisition is synchronously matched with the size of the selected area. When the drone is equipped with data acquisition equipment, the data acquisition module first obtains the distribution coordinates of the water facilities to be inspected, automatically generates a pre-planned flight path covering all target facilities, and dynamically adjusts the flight altitude and shooting angle in real time according to the airspace environment during the flight, so that the appearance images of each water facility cover at least three different perspectives. At the same time, the continuously acquired appearance images are stitched together in real time to form a panoramic appearance image of the overall appearance of the facility and the details of key components. The control module is used to receive operation instructions from inspection personnel, adjust the acquisition range and accuracy of the acquisition module in real time based on the operation instructions, and simultaneously classify and remove redundancy from the two types of acquired data. When the control module adjusts the acquisition range and accuracy of the acquisition module in real time based on operation commands, it follows the following rules: Step 1: The voice commands and hand gesture commands of the inspection personnel are analyzed and correlated. Not only are facility targets (such as pump station motors and pipe welds) and inspection requirements (such as checking for deformation and measuring leakage) extracted from the voice, but the spatial coordinates and movement range of the gesture selection are also matched simultaneously to generate composite control commands that include target objects, spatial range and task priority. Step 2: In terms of range control, based on the historical abnormal data of the facilities to be collected, the potential hazard association range is automatically expanded by 5%-15% on the basis of the area selected by gesture. When the drone is collecting data, it will also dynamically adjust the flight radius according to the "approach" or "detour" intention in the voice command to ensure coverage of key areas while avoiding obstacles. Step 3: In terms of acquisition accuracy control, an adaptation logic is introduced between real-time environmental parameters (light intensity, humidity) and the initial operating status of the facility (such as the real-time speed of the motor). When the light is dim, the low light sensitivity of the image sensor is automatically increased and the anti-interference focal length is matched after the hand gesture command is triggered. When the motor speed is high, the sampling frequency is increased simultaneously (such as from 2 frames / second to 15 frames / second) to capture the appearance details of the slight shaking at the shaft end. Finally, the system will record the operational preferences of the inspection personnel (such as a person who often focuses on inspecting pipeline interfaces). When the person approaches the corresponding facility area without issuing a clear instruction, the system will preset the collection accuracy to the optimal value of the scene in advance (such as 4K resolution and high-frequency sampling mode required for interface detection). The control module performs data classification and redundancy removal, following the following rules: The collected data is divided into primary categories based on appearance status data and operating parameter data. The appearance status data is further divided into secondary categories based on images of the main facility, images of auxiliary components, and environmental images. The operating parameter data is divided into secondary categories based on electrical parameters, mechanical parameters, metering parameters, and signal parameters. Electrical parameters include operating voltage and operating current values, mechanical parameters include operating speed, vibration frequency, and valve opening degree, metering parameters include real-time metering readings, and signal parameters include data transmission signal strength. For data within the same secondary category, calculate the similarity between any two sets of data. When the similarity exceeds a preset similarity threshold, retain the data whose collection time is closer to the current inspection time and delete redundant data. The formula for calculating data similarity is: ; In the formula: These are the weighting coefficients. ∈ (0, 1); The feature similarity between the two sets of appearance status data is determined by calculating the matching degree of SIFT feature points extracted from the images; The numerical similarity between the two sets of operating parameter data; The above formula introduces weighting coefficients to differentiate the impact of appearance status data and operating parameter data on redundancy removal. When appearance status data has a higher impact on the accuracy of redundancy removal, the weighting coefficient is increased, and vice versa, ensuring that the redundancy removal result matches the actual importance of the data. At the same time, the feature similarity of appearance status data is calculated with the help of SIFT feature point matching degree, which can accurately capture the differences in image data. The numerical similarity of operating parameter data is compared with the relationship between the parameter values ​​and the range of normal values, which effectively reflects the consistency of parameter data. The combination of the two can comprehensively and accurately determine whether there is redundancy in the two sets of data, avoiding the redundancy identification bias caused by judging from a single data dimension. Furthermore, weighting coefficients Instead of using fixed values, the system adjusts in real time based on the difference in complexity between the two types of data in the current inspection batch. Specifically, the system calculates the information entropy of the appearance status data and the operating parameter data for that batch. When the information entropy of the appearance image is significantly higher than that of the operating parameters, it indicates that the appearance data contains richer and more effective information, and the entropy is automatically increased. The value makes the redundancy removal judgment focus more on appearance feature matching; conversely, when the information entropy of the running parameters is higher, it reduces the redundancy. The value makes the judgment more dependent on the difference in parameter values; The formula for dynamically adjusting the weighting coefficients is:

[0020] In the formula: The normalized information entropy of the current batch of appearance status data is calculated by extracting grayscale histograms from all appearance images in the batch and applying the information entropy formula. Calculation, where grayscale value The probability of occurrence; To calculate the normalized information entropy of the current batch of operating parameter data, all operating parameter values ​​in the batch are standardized and discretized into 256 intervals. The frequency of each interval is then counted and sorted according to... Calculation, where For the first The frequency of each interval; The hyperbolic tangent function maps the information entropy difference to... The interval guarantees Values ​​in Within the range; This mechanism allows for the use of external data with significantly higher information entropy than parameter data (e.g., rich images of complex pipeline environments but small fluctuations in operating parameters). Automatically approaching 0.8, redundancy removal judgment mainly relies on appearance features; when the entropy of operating parameters is higher (such as frequent changes in facility operating conditions but similar appearance images). Automatically approaching 0.2, the judgment mainly relies on parameter differences, avoiding redundant identification errors caused by fixed weights in different inspection scenarios, and improving the accuracy of data redundancy removal; That is, when the appearance status data has a greater impact on the redundancy removal accuracy of water facility inspection data, The larger the value, the greater the impact of operating parameter data on the accuracy of redundancy removal in water facility inspection data. The smaller the value, , This indicates the number of operating parameters under this secondary category. Let represent the values ​​of the i-th running parameter in each of the two sets of data. This represents the range of the normal value range of the i-th operating parameter; The analysis module is used to receive data that has been classified and deredundant processed in the control module, identify abnormal characteristics of water facilities based on the received data, and determine the suspected abnormal types of water facilities. The analysis module identifies abnormal features of water facilities to determine suspected anomaly types as follows: Feature extraction is performed on the appearance status data processed by the control module to obtain appearance feature parameters including at least the proportion of damaged area on the facility surface, the proportion of pixels in color abnormal areas, and the degree of component deformation. The processed operating parameter data is then standardized to obtain standardized electrical parameters, mechanical parameters, metering parameters, and signal parameters. The appearance feature parameters are combined with the standardized operating parameters to form the feature vector of the water facility. Where m is the total number of feature parameters, This is the quantized value of the first feature parameter; Obtain historical normal inspection data for a specified time interval. Based on this data, calculate the normal mean and standard deviation for each characteristic parameter. Then, determine the facility anomaly level based on the normal mean and standard deviation. , This represents the weight coefficient of the k-th feature parameter. This represents the quantized value of the k-th feature parameter. This represents the normal mean and normal standard deviation. Used to represent the deviation of the k-th feature parameter; The above formula integrates the appearance feature parameters and standardized operating parameters into a feature vector, covering multi-dimensional information on facility appearance and operation, providing a comprehensive data foundation for anomaly calculation. Then, the normal mean and standard deviation of each feature parameter are obtained using historical normal inspection data, which serve as the benchmark for judging whether the parameter is abnormal, ensuring that the anomaly judgment conforms to the normal operation law of the facility. By setting weight coefficients, more weight is given to feature parameters that have a higher impact on the safety of facility operation, so that the anomaly calculation results can better reflect the impact of key parameter anomalies on facility safety. Finally, the anomaly degree is obtained by summing the products of the deviation degree of each feature parameter and the corresponding weight, providing an accurate basis for subsequent suspected anomaly judgment. when When the deviation exceeds the preset abnormal threshold, the facility is judged to have a suspected abnormality. At the same time, according to the deviation of each characteristic parameter, the top three characteristic parameters with the largest deviation are selected. Then, according to the preset feature-to-abnormality type mapping relationship, the suspected abnormality type of the water facility is determined. For example, when the vibration frequency deviation and the operating speed deviation are the top two deviations, the suspected abnormality type is judged to be mechanical transmission abnormality. The standardization of operational parameter data involves converting operational parameter values ​​with different dimensions and numerical ranges into quantified values ​​of a unified dimension based on the normal range or historical normal data for each operational parameter. >0, and The higher the degree of impact of the k-th characteristic parameter on facility operation safety, the better. The larger the value, the better; conversely, the smaller the value, the better. The smaller the value; Furthermore, to enhance the ability of anomaly calculation to identify facility deterioration trends, the system introduces a time-series trend correction mechanism. Specifically, after extracting the feature vector at the current inspection time, the system automatically obtains the feature vectors of the facility's three most recent consecutive historical inspections, calculates the time-series change trend of each feature parameter, and applies an additional trend penalty coefficient to parameters showing a continuous deterioration trend, making the anomaly results more accurately reflect the facility's deterioration risk. The revised formula for calculating anomaly is:

[0021] In the formula: The corrected facility anomaly level; For the first Trend sensitivity coefficients of each characteristic parameter, with a range of values. For key parameters that affect facility safety (such as vibration frequency and component deformation), larger values ​​(such as 0.8) should be taken, and for minor parameters, smaller values ​​(such as 0.2) should be taken. For the first The time-series degradation index of each feature parameter is calculated using the following formula:

[0022] in For the current inspection time The quantized value of each feature parameter, , , These are the quantified values ​​of this parameter from the three most recent consecutive historical inspections. This represents the normal standard deviation of the parameter. When When this occurs, it indicates that the parameter shows a deteriorating trend of continuously deviating from the normal mean during multiple consecutive inspections, at which point the correction term... When the value is greater than 1, the contribution of this parameter to the anomaly score is amplified; when... When the parameter does not show a continuous deterioration trend, the correction term degenerates to 1, which does not affect the original anomaly calculation. It should be noted that when the facility has fewer than 3 historical inspection records, the time-series trend correction mechanism will not be activated and the correction item will be reduced to 1. The coordination module is used to create a multimodal information interaction channel between inspection personnel and remote technicians, and to transmit data collected by the acquisition module and processed by the control module, as well as analysis results from the analysis module. The multimodal information interaction channels created by the coordination module include voice interaction channels, image sharing channels, and parameter interaction channels; When the analysis module determines that the facility has a suspected anomaly and the anomaly level exceeds the preset high-risk threshold, the coordination module automatically increases the transmission frame rate of the image sharing channel to the preset high-frequency range, switches the voice interaction channel to low-latency mode with the latency controlled within the preset threshold, and suspends the update of non-related parameters in the parameter interaction channel. When a remote technician initiates an interactive command through any channel, the other two channels automatically synchronize the timestamp and associated object of that command. An example is provided below: When technicians mark damaged areas in the image sharing channel, the voice interaction channel automatically records the voice segment at the moment of marking, and the parameter interaction channel simultaneously highlights the corresponding operating parameter curve segment of the area. The recording module is used to encrypt and store the collected data, operation instructions, and anomaly analysis results, generate a unique inspection record identifier, and associate the inspection time and inspection personnel information of the corresponding facilities. When the recording module performs encrypted storage and unique identifier generation, it follows the following rules: Symmetric encryption algorithms are used to encrypt the collected data, operation instructions, and anomaly analysis results. The encryption key is generated by combining the pre-set unique identification key of the inspector and the facility-specific key. The inspector identification key is bound to the inspector's fingerprint or face, and the facility-specific key is a pre-set unique key for each water facility. The encrypted data is stored in a distributed database, and the access permissions of the database nodes correspond one-to-one with the job permissions of the inspectors. After the collected data, operation instructions, and anomaly analysis results are encrypted and stored in the distributed database, the corresponding cached data in the source module of the collected data, operation instructions, and anomaly analysis results are all deleted. The unique inspection record identifier follows the following rules when generated: ,in A unique code indicating a water facility. Indicates the timestamp of the inspection operation. This indicates the unique employee number of the patrol inspector. The SHA-256 hash function is used to generate a unique inspection record identifier, which is then associated with the inspection time, inspection personnel information, and corresponding facility code for storage. The above formula uses the unique code of water facilities, the inspection timestamp accurate to the millisecond, and the unique employee number of the inspector as the basic data. These three types of data ensure the unique correspondence between the identifier and the facility, the inspection time, and the inspector, respectively, avoiding the confusion of identifiers due to data ambiguity. It not only ensures the uniqueness of the inspection record identifier and prevents the identifier from being duplicated, but also effectively protects the security of basic data information and prevents the leakage of key information. At the same time, the generated unique identifier is associated with the inspection time, the inspector information, and the facility code for storage, which further strengthens the binding between the identifier and the inspection-related information, making it easier to quickly query and trace the inspection record in the future. Among them, the initial settings Accurate to the millisecond level; The optimization module is used to generate and set the optimal path for the next inspection based on the historical inspection data stored in the recording module, combined with the distribution location and operating status of water facilities. The process for the optimization module to generate the optimal path for the next inspection is as follows: Based on the historical inspection data stored in the recording module, an inspection weight is assigned to each water facility. The weight is calculated as the product of the number of suspected anomalies in history, the functional importance of the facility, and the time since the last inspection. If any one or more of these values ​​are zero, then the zero value is not included in the product calculation. This setting ensures that the weights accurately reflect the actual inspection priorities of the facilities, providing a reasonable and reliable basis for the subsequent optimal inspection path ranking, and ensuring that inspection resources are prioritized for high-risk and high-importance facilities. Synchronously set path constraints: the inspection path should not repeatedly pass through the same area; Finally, the deployment locations of each water facility are used as path nodes. Based on the inspection weight of each water facility, the corresponding path nodes are sorted from high to low, and the path nodes are connected in order according to the sorting results, and the connection results meet the set path constraints. When the connecting path between any two path nodes is not unique, the shorter path is selected first, and the remaining paths are used to satisfy the path constraint conditions. Among them, the appearance status data and operating parameter data of water facilities are collected by inspection personnel or drones equipped with corresponding data collection equipment in a non-contact manner. The appearance status data of water facilities includes at least the appearance image of the water facilities, and the operating parameter data includes all quantitative parameters reflecting the functional operation status of the water facilities, such as: temperature, working voltage, working current value, operating speed, vibration frequency, valve opening degree, real-time metering reading, data transmission signal strength, etc. The operation instructions of the inspection personnel include voice instructions such as "open valve" and hand gesture instructions such as gesture selection of pipes. The data acquisition module is interconnected with the control module via a wireless network. The control module is interconnected with the analysis module via a wireless network. The analysis module is interconnected with the coordination module via a wireless network. The coordination module is interconnected with the recording module and the optimization module via a wireless network. The recording module is interconnected with the optimization module via a wireless network.

[0023] In this embodiment, the acquisition module collects and stores the appearance status data and operating parameter data of the water facilities. The control module receives the operation instructions from the inspection personnel and adjusts the acquisition range and accuracy of the acquisition module in real time based on the operation instructions. It also classifies and removes redundancy from the two types of data simultaneously. The analysis module receives the data that has been classified and deredundanted from the control module and identifies the abnormal characteristics of the water facilities based on the received data. It determines the suspected abnormal type of the water facilities. The coordination module further creates a multimodal information interaction channel between the inspection personnel and remote technicians, transmitting the data collected by the acquisition module and processed by the control module, as well as the analysis results from the analysis module. The recording module encrypts and stores the acquired data, operation instructions, and abnormal analysis results, generating a unique inspection record identifier and associating it with the inspection time and inspection personnel information of the corresponding facilities. Finally, the optimization module generates and sets the optimal path for the next inspection based on the historical inspection data stored in the recording module, combined with the distribution location and operating status of the water facilities.

[0024] The system in the above embodiments accurately collects water facility data in a non-contact manner, optimizes collection accuracy and efficiency, reduces data redundancy, and can quickly identify facility anomalies and clarify suspected types, facilitating inspection and remote personnel collaboration. At the same time, data encryption storage ensures security, and the system can plan the optimal inspection route to avoid repeated areas, prioritize key facilities, save time and costs, effectively improve inspection quality, and ensure the stable operation of water facilities.

[0025] Example 2: At the implementation level, based on Example 1, this example refers to... Figure 2 A further detailed description of the intelligent water facility inspection system based on multimodal interaction in Example 1 is provided below: A smart inspection method for water facilities based on multimodal interaction includes: The inspection personnel use handheld data collection devices or drones equipped with data collection devices to collect data on the appearance and operating parameters of water facilities in a non-contact manner, and then initially store the collected data. The system receives operation instructions from inspection personnel to adjust the collection range and accuracy of the data acquisition module, classifies the collected data into primary categories based on appearance and operating parameters, further subdivides them into secondary categories, and then deletes redundant data based on the data similarity calculation formula. The appearance feature parameters of the extracted and processed data are combined with the standardized operating parameters to form a feature vector. The facility anomaly degree is calculated. If the anomaly degree exceeds the preset threshold, the suspected anomaly type of the water facility is determined according to the deviation of the feature parameters. Create multimodal interactive channels for voice, image, and parameters to transmit data and analysis results; optimize channel transmission modes in case of high-risk anomalies; and synchronize the timestamps and associated objects of interactive commands for each channel. The data is symmetrically encrypted using a combination of the inspector's identity key and the facility's exclusive key, and stored in a distributed database. This generates a unique inspection record identifier containing the facility code, timestamp, and employee number, which is then stored in association. The facility inspection weight is calculated based on the number of suspected anomalies in history, the importance of facility functions, and the time since the last inspection. The path nodes are sorted by weight and combined with the constraints to generate the next optimal inspection path.

[0026] In summary, during the execution of the system and method in the above embodiments, the inspection personnel use handheld devices or drones to collect the appearance and operation data of water facilities in a non-contact manner. The collection focus, frequency, and flight path are adjusted in real time to ensure comprehensive and accurate data. The collected data is classified and redundancy is removed to improve quality. The abnormality degree of the facilities is calculated by combining historical normal data, and abnormal characteristics are accurately identified and suspected abnormality types are determined. A multimodal information interaction channel is established. In the case of high-risk abnormalities, the transmission mode is optimized to improve the collaboration efficiency between inspection and remote technical personnel. The data is encrypted and stored and a unique inspection identifier is generated to ensure security and traceability. The optimal path is planned based on historical inspection data and facility weights to avoid duplication and prioritize coverage of important facilities. Overall, the level of intelligence, operation efficiency, and safety assurance capability of water facility inspection are improved.

[0027] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-modal interaction based intelligent inspection system for water utility facilities, characterized in that, The utility model relates to a kind of water affairs facility inspection system and method, including: Collecting module, for collecting the appearance state data of water affairs facility, operating parameter data, the data collected are stored; Control module, for receiving the operation instruction of inspection personnel, based on operation instruction, the collection range and accuracy of collecting module are controlled in real time, the two kinds of data collected are classified and de-redundancy processed synchronously; Analysis module, for receiving the data classified and de-redundancy processed in control module, based on received data, identify the abnormal characteristics of water affairs facility, determine water affairs facility suspected abnormal type; Coordination module, for creating the multimodal information interaction channel of inspection personnel and remote technical personnel, transmit the data collected by collecting module and the analysis result after processing by control module; Recording module, for the encrypted storage of collected data, operation instruction, abnormal analysis result, generate unique inspection record identification, associate corresponding facility's inspection time and inspection personnel information; Optimization module, for generating and setting the optimal path of next inspection according to the historical inspection data stored by recording module, combined with the distribution location and operating state of water affairs facility. The appearance state data and operating parameter data of water affairs facility are collected by inspection personnel or unmanned aerial vehicle with corresponding collecting equipment in a non-contact manner.

2. The multi-modal interaction based intelligent inspection system for water utility facilities as claimed in claim 1 wherein, When the collecting module performs non-contact collection, the collecting module automatically adjusts the focal length and sampling frequency of the collecting equipment according to the hand posture instruction of the inspection personnel when the inspection personnel collects by hand-held collecting equipment. When the inspection personnel selects a specific area of the pipeline by hand posture instruction, the sampling frequency of the collecting equipment is automatically increased to a preset high frequency interval, and the appearance image collection resolution is synchronized to match the size of the selected area. When the unmanned aerial vehicle carries the collecting equipment, the collecting module first obtains the distribution coordinates of the water affairs facility to be inspected, automatically generates a pre-planned flight path covering all target facilities, and dynamically adjusts the flight height and shooting angle in real time during flight according to the airspace environment, so that the appearance image collection of each water affairs facility covers at least three different angles. At the same time, the continuously collected appearance images are real-time spliced to form a panoramic appearance image of the overall appearance and key component details of the facility.

3. The multi-modal interaction based intelligent inspection system for water utility facilities as claimed in claim 1 wherein, The control module performs data classification and de-redundancy processing stage, subject to: The collected data is divided into primary categories according to appearance state data and operating parameter data, and then the appearance state data is divided into secondary categories according to facility main body image, accessory component image and environment image, and the operating parameter data is divided into secondary categories according to electrical parameters, mechanical parameters, measurement parameters and signal parameters; For data of the same secondary category, calculate the similarity of any two groups of data. When the similarity is greater than a preset similarity threshold, keep the data with a collection time closer to the current inspection time, and delete the redundant data. The data similarity calculation formula is: ； In the formula: is a weight coefficient, ∈(0, 1); is the feature similarity of the two sets of appearance state data, which is determined by the SIFT feature point matching degree calculation of the extracted image; is the numerical similarity of the two sets of running parameter data; wherein, , represents the number of operating parameters under the secondary category, respectively represent the value of the i-th operating parameter in the two groups of data, represents the range of the normal value range of the i-th operating parameter.

4. The multi-modal interaction based intelligent inspection system for water utility facilities as claimed in claim 1 wherein, The operation of the analysis module to identify the abnormal characteristics of water affairs facility to determine the suspected abnormal type is: Feature extraction is performed on the appearance state data processed by the control module to obtain appearance feature parameters including at least facility surface damage area ratio, color abnormal area pixel ratio and component deformation degree. The processed operation parameter data is standardized to obtain standardized electric parameters, mechanical parameters, metering parameters and signal parameters, the appearance characteristic parameters are combined with the standardized operation parameters to form a characteristic vector of the water facilities wherein m is the total number of characteristic parameters, is the quantized value of the first characteristic parameter Obtaining historical normal inspection data of a specified time interval, based on the historical normal inspection data, statistically obtaining a normal mean and a normal standard deviation of each feature parameter, and obtaining a facility anomaly degree based on the normal mean and the normal standard deviation , denotes a weight coefficient of the kth feature parameter, denotes a quantized value of the kth feature parameter, denotes a normal mean and a normal standard deviation, for representing a deviation degree of the kth feature parameter; When When the deviation degree is greater than the preset abnormal threshold, it is determined that the facility has a suspected abnormality, and the first three feature parameters with the greatest deviation degrees are selected according to the deviation degrees of the feature parameters. Then, the suspected abnormal type of the water facility is determined according to the preset mapping relationship between the feature and the abnormal type.

5. The multi-modal interaction based intelligent inspection system for water utility facilities as claimed in claim 1 wherein, The multi-modal information interaction channel created by the coordination module includes a voice interaction channel, an image sharing channel, and a parameter interaction channel. When the analysis module determines that the facility has a suspected abnormality and the abnormality degree exceeds the preset high-risk threshold, the coordination module automatically raises the transmission frame rate of the image sharing channel to a preset high-frequency range, switches the voice interaction channel to a low-delay mode, controls the delay within a preset threshold, and suspends the update of non-associated parameters in the parameter interaction channel. When the remote technician initiates an interaction instruction through any channel, the other two channels automatically synchronize the timestamp and associated object of the instruction.

6. The multi-modal interaction based intelligent inspection system for water utility facilities as claimed in claim 1 wherein, The record module performs encryption storage and unique identification generation in compliance with the following: The collected data, operation instructions, and abnormality analysis results are encrypted using a symmetric encryption algorithm, and the encryption key is generated by combining a preset inspection personnel unique identity key and a facility exclusive key. The inspection personnel identity key is bound to the inspection personnel's fingerprint or face, and the facility exclusive key is a unique key preset for each water service facility. The encrypted data is stored in a distributed database, and the access permissions of the database nodes correspond to the post permissions of the inspection personnel. After the collected data, operation instructions, and abnormality analysis results are encrypted and stored in the distributed database, the corresponding cache data in the collected data, operation instructions, and abnormality analysis results source module are deleted. The unique inspection record identification is generated in compliance with the following: wherein represents a unique code of the water utility facility, represents a timestamp of the inspection operation, represents a unique worker number of the inspection personnel, represents a SHA-256 hash function, and the newly generated unique inspection record identifier is stored in association with the inspection time, the inspection personnel information, and the corresponding facility code.

7. The multi-modal interaction based intelligent inspection system for water utility facilities as claimed in claim 1 wherein, The optimization module runs the following process to generate the optimal path for the next inspection: Based on the historical inspection data stored by the record module, configure an inspection weight for each water service facility. The weight calculation value is the product of the historical suspected abnormality times, the functional importance of the facility, and the time since the last inspection. If any of the values is zero, the zero term is not included in the product operation. Synchronize the path constraint conditions: the inspection path does not repeat the same area. Finally, use the deployment location points of each water service facility as path nodes, sort the corresponding path nodes based on the inspection weights from high to low, and connect the path nodes in order according to the sorting results. The connection result meets the set path constraint conditions. If there are multiple paths between any two path nodes, prefer the short path, and use the remaining paths to meet the path constraint conditions.

8. The multi-modal interaction based intelligent inspection system for water utility facilities as claimed in claim 1 wherein, The collection module is connected to the control module through a wireless network, the control module is connected to the analysis module through a wireless network, the analysis module is connected to the coordination module through a wireless network, the coordination module is connected to the record module and the optimization module through a wireless network, and the record module is connected to the optimization module through a wireless network.

9. A method for intelligent inspection of water facilities based on multi-modal interaction, the method being a method for implementing the system for intelligent inspection of water facilities based on multi-modal interaction according to any one of claims 1-8, characterized in that, It includes: Using a non-contact method such as a handheld collection device or a drone-mounted collection device, collect the appearance state data and operating parameter data of the water service facility, and perform initial storage of the collected data. Receive the operation instructions of the inspection personnel to control the collection range and accuracy of the collection module, divide the collected data into primary categories based on appearance and operating parameters, and further subdivide them into secondary categories. Delete redundant data based on a data similarity calculation formula. The appearance characteristic parameters of the processed data form a characteristic vector with the standardized operation parameters, the facility anomaly degree is calculated, and if the anomaly degree exceeds a preset threshold, the suspected abnormal type of the water facility is determined according to the characteristic parameter deviation degree; A voice, image and parameter multi-modal interaction channel is created to transmit data and analysis results, and the transmission mode of the channel is optimized when a high-risk anomaly occurs, and the time stamps and associated objects of the interaction instructions of each channel are synchronized; The data is symmetrically encrypted by a key generated by combining the identity key of the inspection personnel and the exclusive key of the facility, and is stored in a distributed database, and a unique inspection record identifier containing the facility code, time stamp and personnel number is generated and stored in association; The facility inspection weight is calculated according to the number of historical suspected anomalies, the importance of the facility function and the time since the last inspection, the path nodes are sorted by weight, and the next optimal inspection path is generated in combination with the constraint conditions.

Images