A park equipment inspection exception identification and intelligent interaction method and device

Abstract

The application discloses a kind of park equipment inspection exception identification and intelligent interaction method and device, comprising: obtaining environmental perception data, environmental perception data includes image data, audio data and text data;Environmental perception data is input into park equipment intelligent inspection model, and park equipment intelligent inspection model outputs feedback result;Park equipment intelligent inspection model executes steps: image data, audio data and text data are respectively preprocessed;Processed image data, audio data and text data are input into cross-modal fusion module, and output exception category and risk score;Intelligent feedback result is generated according to exception category and risk score;Feedback result is dynamically adjusted according to inspection scene and task stage;The application realizes the exception identification and intelligent interaction for the inspection scene of park equipment, and through the cooperation of three types of data of image, audio, text, the multi-dimensional state of equipment vision, acoustics, semantics is comprehensively covered.

Description

Technical Field

[0001] This invention relates to the field of intelligent equipment inspection technology, and in particular to a method and device for identifying and intelligently interacting with abnormal equipment inspections in industrial parks. Background Technology

[0002] As the core carrier of industrial production, energy supply, and municipal operation and maintenance, industrial parks deploy a large number of critical equipment, whose operational status directly affects the park's safe production, energy efficiency, and operation and maintenance costs. Traditional park equipment inspections mainly rely on manual on-site operations. Inspectors complete inspection tasks by visually observing the equipment's appearance, reading instrument readings, recording on-site sounds, and filling out paper inspection forms. This model has many inherent defects: on the one hand, manual inspections are affected by subjective factors such as personnel's professional ability, sense of responsibility, and fatigue, which can easily lead to missed inspections, false inspections, and non-standard records, making it impossible to guarantee the consistency and accuracy of inspection results; on the other hand, manual inspections are inefficient and costly, making it difficult to achieve high-frequency, comprehensive, and routine inspections. They also lack the ability to provide early warnings of hidden equipment failures and sudden anomalies, which can easily lead to serious consequences such as equipment downtime and safety accidents.

[0003] With the development of artificial intelligence technology, intelligent inspection solutions are gradually being applied in industrial scenarios. For example, single-vision-based equipment appearance inspection uses target detection models, segmentation models, or defect recognition models to identify visual problems such as meter status, leaks, damage, and abnormal indicator lights. Inspection assistance systems based on acoustic monitoring or voice question-and-answer use microphones, vibration sensors, or voice assistants to identify abnormal sounds from equipment, or upload the inspector's voice questions to a voice recognition and question-and-answer module to output text results or voice broadcasts. However, existing technologies have significant limitations: insufficient robustness of single-modal approaches, difficulty in unifying decisions based on multi-source heterogeneous data, semantic disconnect between anomaly identification results and inspection tasks, lengthy and fragmented on-site feedback, and insufficient real-time performance in edge deployments. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that the existing technology suffers from insufficient robustness of single acoustic judgment, difficulty in unified decision-making based on multi-source heterogeneous data information, and low efficiency.

[0005] To address the aforementioned technical problems, this invention provides a method and apparatus for identifying and intelligently interacting with abnormalities in park equipment inspections.

[0006] In a first aspect, the present invention provides a method for identifying and intelligently interacting with abnormalities in park equipment inspections, comprising:

[0007] Acquire environmental perception data of the park, including image data, audio data, and text data;

[0008] The environmental perception data is input into the intelligent inspection model of park equipment, and the intelligent inspection model of park equipment outputs feedback results.

[0009] The intelligent inspection model for park equipment executes the following steps:

[0010] The image data, audio data, and text data are preprocessed respectively;

[0011] The processed image, audio, and text data are input into the cross-modal fusion module, which outputs anomaly categories and risk scores.

[0012] Intelligent feedback results are generated based on the anomaly category and risk score;

[0013] The feedback results are dynamically adjusted based on the inspection scenario and task stage.

[0014] Through a full-link architecture of multimodal data input, cross-modal fusion, intelligent feedback, and dynamic strategies, an end-to-end intelligent inspection closed loop is constructed. By coordinating image, audio, and text data, it comprehensively covers the visual, acoustic, and semantic multi-dimensional status of the equipment.

[0015] The preprocessing includes:

[0016] After performing denoising, white balance correction, brightness normalization, resolution adjustment and target region cropping on the image data, an image input suitable for encoder processing is obtained. The image input is then converted into visual features by a visual encoder.

[0017] After performing endpoint detection, noise suppression, gain control, and short-time spectrum transformation on the audio data, audio features are extracted.

[0018] After performing word segmentation, removal of redundant stop words, key entity recognition, and sequence concatenation on the text data, the user's voice input recognition model is converted into speech text. The speech text, along with the OCR text and task text in the text data, are constructed into a unified text sequence, which is then input into the text encoder to form text features.

[0019] This invention achieves unified semantic modeling through visual encoding, text encoding, and cross-modal attention fusion, enabling image evidence, audio evidence, and text evidence to be cross-verified, thereby reducing the probability of false positives and false negatives.

[0020] The cross-modal fusion module performs the following steps:

[0021] Perform dimensional unification and semantic alignment on the aforementioned visual features, audio features, and text features;

[0022] Based on the aligned features, calculate the cross-modal attention weights and generate the fused output, as shown in the following formula:

[0023]

[0024]

[0025] in, This is the attention weight matrix. The Softmax activation function is used. For query vector, For key vectors, This is the transpose of the key vector. The dimension of the key vector. It is a square root function. For attention output, It is a value vector;

[0026] The outputs of the cross-modal attention layers are integrated into a unified fusion feature. The formula is as follows:

[0027]

[0028] in, As a feature of fusion, For feature fusion function, As a visual feature, For audio features, Text features;

[0029] The fusion features Input the anomaly classification header and risk scoring header, and output the anomaly categories respectively. and risk score The formula is as follows:

[0030]

[0031]

[0032] in, As an anomaly category, For anomaly classifier, As a feature of fusion, As a risk score, As the weight for risk scoring, It is a feedforward network. This is for risk scoring bias.

[0033] A multimodal joint modeling method combining visual, speech, OCR text, and task text is adopted to enable device appearance, operating sound, and text semantics to collaboratively determine anomalies in a unified semantic space.

[0034] The image data prioritizes locating the instrument panel area, valve area, wiring area, nameplate area, and warning sign area.

[0035] The audio data is used to prioritize the identification of key acoustic events, including abnormal motor noises, leakage sounds, friction sounds, and alarm beeps.

[0036] The text data is extracted first from the speech transcription results, OCR recognition results, and task work orders, including the equipment name, inspection location, abnormal keywords, and handling action keywords.

[0037] The intelligent feedback result includes at least one of the following: abnormal object, abnormal location, abnormality level, and suggested action.

[0038] The step of generating intelligent feedback results based on the anomaly category and risk score includes:

[0039] The abnormal location and inspection task context are obtained based on the image data and text data. The abnormal location includes one or more of the following: equipment number, component name, area name, or image positioning area. The inspection task context includes one or more of the following: task work order information, equipment type, inspection round, historical abnormal records, or current inspection stage.

[0040] Based on the abnormality categories of the park inspection, the suggested actions are preset, including one or more of the following: review, record, shutdown inspection, power outage, component replacement, or reporting for maintenance.

[0041] Based on the anomaly category, risk score, inspection task context, anomaly location, and suggested action, a set of feedback semantic slots is constructed. The formula is as follows:

[0042]

[0043] in, To provide a set of semantic slots for feedback, Functions are built to provide feedback semantic slots. As an anomaly category, As a risk score, This is an abnormal location. For suggested actions, For the context of the inspection task;

[0044] The feedback priority is determined based on the risk score, as follows:

[0045]

[0046] in, As a feedback priority, Determine the function for priority. Risk score;

[0047] Based on the set of feedback semantic slots and feedback priority, a corresponding template is selected from the preset feedback template set and candidate feedback text is generated:

[0048]

[0049] in, For candidate feedback text, For template matching and filling functions, To provide a set of semantic slots for feedback, Prioritize feedback;

[0050] The candidate feedback text is processed by terminology normalization and redundancy reduction to obtain the final feedback text:

[0051]

[0052] in, For the final feedback text, For text normalization functions, For candidate feedback text;

[0053] Voice control parameters are generated based on the final feedback text, feedback priority, and environmental noise parameters.

[0054]

[0055] in, Here, N represents the voice control parameter, and N represents the ambient noise parameter. For the final feedback text, As a feedback priority, Function for generating voice control parameters;

[0056] The set of voice control parameters include

[0057]

[0058] in, For voice control parameter set, For speech rate parameters, For volume parameters, For pitch parameters, For pause parameters;

[0059] After prosodic annotation of the final feedback text, it is input into the speech synthesis module to generate a speech feedback signal:

[0060]

[0061]

[0062] in, This represents the text to be synthesized with prosodic annotations. This represents the prosody annotation function. For the final feedback text, Here is the set of voice control parameters, and F is the voice feedback signal. This is a speech synthesis function;

[0063] Synchronously output graphical prompts, including anomaly boxes, risk colors, device numbers, and suggested action buttons.

[0064] The multi-factor collaborative feedback design enables inspection personnel to quickly locate anomalies, identify risks, and take appropriate measures, thereby improving the efficiency and accuracy of anomaly handling.

[0065] The feedback priority is divided into low priority, medium priority, and high priority based on the risk score, as expressed below:

[0066]

[0067] in, As a feedback priority, Low priority As a risk score, Medium priority As a high priority, The risk score threshold is one. The risk score threshold is two.

[0068] The dynamic adjustment of feedback results based on the inspection scenario and task stage includes:

[0069] When the noise in the inspection area exceeds the set noise threshold, the weights of the visual and OCR results are increased by the set weight step size.

[0070] When the screen is obstructed or the light intensity is lower than the set light intensity threshold, the constraint weights of audio and task text are increased by the set weight step size.

[0071] When a high-risk anomaly is detected, high-priority voice and visual cues will be output first.

[0072] Once the inspection personnel confirm the anomaly or complete the review, they will update the system's anomaly status and prompting policy.

[0073] It also includes: fine-tuning and online optimization of the parameters of the intelligent inspection model for park equipment based on historical inspection data, manual annotation results, and user operation feedback;

[0074] The parameter update formula for fine-tuning is:

[0075]

[0076] in, for, These are the current model parameters. For learning rate, for, This is the total loss function;

[0077] The total loss function is composed of anomaly classification loss, text generation loss, and cross-modal alignment loss:

[0078]

[0079] in, For the total loss function, This represents the anomaly classification loss. This indicates the loss in feedback text generation. This represents the alignment loss between visual, audio, and text features. The weighting coefficient is one. The weighting coefficient is two. The weighting coefficient is three.

[0080] In the dynamic adjustment phase, a scenario-adaptive weight adjustment and model fine-tuning mechanism is introduced to enable the system to adapt to the actual needs of different parks, different equipment and different task stages.

[0081] Secondly, the present invention provides a park equipment inspection anomaly identification and intelligent interaction device, comprising:

[0082] The environmental perception module is used to acquire environmental perception data of the park, including image data, audio data, and text data.

[0083] The inspection feedback module is used to input the environmental perception data into the intelligent inspection model of park equipment, and the intelligent inspection model of park equipment outputs feedback results.

[0084] The intelligent inspection model for park equipment executes the following steps:

[0085] The image data, audio data, and text data are preprocessed respectively;

[0086] The processed image, audio, and text data are input into the cross-modal fusion module, which outputs anomaly categories and risk scores.

[0087] Intelligent feedback results are generated based on the anomaly category and risk score;

[0088] The feedback results are dynamically adjusted based on the inspection scenario and task stage.

[0089] Compared with existing technologies, the beneficial effects achieved by this invention are as follows: This invention realizes anomaly identification and intelligent interaction for park equipment inspection scenarios through a complete technical chain of image processing, speech recognition, text encoding, cross-modal fusion, anomaly scoring, intelligent feedback, and fine-tuning optimization. Through the synergy of image, audio, and text data, it comprehensively covers the visual, acoustic, and semantic multi-dimensional status of equipment. Attached Figure Description

[0090] Figure 1 This is a schematic diagram of the park equipment inspection anomaly identification and intelligent interaction method according to Embodiment 1 of the present invention;

[0091] Figure 2 This is a schematic diagram of the image data processing flow shown in Embodiment 1 of the present invention;

[0092] Figure 3 This is a schematic diagram of the speech recognition and text processing flow shown in Embodiment 1 of the present invention;

[0093] Figure 4 This is a schematic diagram of cross-modal fusion and anomaly scoring as shown in Embodiment 1 of the present invention;

[0094] Figure 5 This is a schematic diagram of intelligent feedback and dynamic adjustment as shown in Embodiment 1 of the present invention;

[0095] Figure 6 This is a block diagram of the overall structure of the park equipment inspection anomaly identification and intelligent interaction system shown in Embodiment 4 of the present invention. Detailed Implementation

[0096] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0097] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0098] Example 1

[0099] This embodiment provides a method for identifying and intelligently interacting with abnormal equipment inspections in a park, including:

[0100] Acquire environmental perception data of the park, which includes image data, audio data, and text data;

[0101] The environmental perception data is input into the park equipment intelligent inspection model, and the park equipment intelligent inspection model outputs feedback results.

[0102] The execution steps of the intelligent equipment inspection model in the park are as follows:

[0103] S1. Preprocess the image data, audio data, and text data respectively;

[0104] S2. Input the processed image data, audio data, and text data into the cross-modal fusion module, and output the anomaly category and risk score;

[0105] S3. Generate intelligent feedback results based on anomaly category and risk score;

[0106] S4. Dynamically adjust feedback results based on inspection scenarios and task stages.

[0107] The preprocessing of environmental perception data in S1 includes:

[0108] S11. After performing denoising, white balance correction, brightness normalization, resolution adjustment and target region cropping on the image data, an image input suitable for encoder processing is obtained. The image input is converted into visual features by the visual encoder.

[0109] S12. After performing endpoint detection, noise suppression, gain control and short-time spectrum transformation on the audio data, extract the audio features;

[0110] S13. After performing word segmentation, removal of redundant stop words, key entity recognition and sequence concatenation on the text data, the user's voice input is converted into speech text by the recognition model. The speech text, OCR text and task text are constructed into a unified text sequence, which is then input into the text encoder to form text features.

[0111] The execution of the intelligent inspection model for park equipment includes the following steps:

[0112] Step 1: Process the image data from the collected environmental perception data, such as... Figure 1 As shown, the steps for identifying and intelligently interacting with abnormal equipment inspections in the park are as follows:

[0113] Step 1.1: As Figure 2 As shown, the input image is processed by the image preprocessing module. The image is then denoised, its brightness normalized, and its size adjusted to obtain the preprocessed image. , is represented as:

[0114]

[0115] in, The original input image, This indicates a noise reduction operation. Indicates size adjustment. This indicates normalization processing. The image is after preprocessing;

[0116] Step 1.2: Input the preprocessed image into the visual encoder to obtain the visual feature vector. :

[0117]

[0118] in, For visual encoder functions, For image feature vectors, The preprocessed image is used to characterize the device's appearance, component locations, and abnormal region features. The visual encoder consists of convolutional layers or visual Transformer layers; if convolutional layers are used, the local feature extraction process is represented as:

[0119]

[0120] in, For image feature vectors, The image after preprocessing. For convolution kernel parameters, For convolution operations, This is a bias term.

[0121] Step 1.3: Extract key regions from the image, including the dashboard area, valve area, wiring area, nameplate area, and warning sign area. If a certain area is denoted as... Then, its OCR text extraction result can be expressed as:

[0122]

[0123] in, This is a text recognition function. To identify the obtained text content, such as equipment number, meter reading, and warning text.

[0124] Step 2 involves converting the collected environmental perception data into text for processing, including the following steps:

[0125] Step 2.1: Prioritize the identification of key acoustic events based on audio data, including abnormal motor noises, leakage sounds, friction sounds, and alarm beeps, such as... Figure 3 As shown, the acquired audio A is preprocessed, including speech endpoint detection, noise suppression, and spectral transformation, to obtain spectral features. Its short-time Fourier transform (STFA) is expressed as:

[0126]

[0127] in, This is the result of the short-time Fourier transform. The original audio signal. For window functions, For frequency, For time, For integration time variable, It is a natural constant.

[0128] Step 2.2: Input the audio features into the speech recognition model to obtain speech transcription. :

[0129]

[0130] in, For speech recognition functions, These are text instructions or verbal inspection records obtained from user voice. These are audio spectral characteristics.

[0131] Step 2.3: Transcribe the speech into text. , text and inspection tasks The text is then concatenated to form a unified text sequence. :

[0132]

[0133] in, To unify the text sequence, for, For speech-to-text transcription, for text, For inspection tasks.

[0134] Step 2.4: Input the unified text sequence into the text encoder to obtain the text semantic vector. :

[0135]

[0136] in, For text semantic vectors, For text encoder functions, To unify the text sequence.

[0137] Step 3: Perform cross-modal fusion and anomaly scoring on the preprocessed data:

[0138] Step 3.1: As Figure 4 As shown, visual features Audio features and text features Input the cross-modal fusion module to obtain fused features :

[0139]

[0140] in, As a feature of fusion, For feature fusion function, As a visual feature, For audio features, For text features, This is achieved using a cross-modal attention mechanism, where:

[0141]

[0142] in, As a feature of fusion, For cross-modal attention functions, As a visual feature, For audio features, Text features; This indicates that text features are used as query vectors, and visual and audio features are semantically aligned.

[0143] Cross-modal attention weights are represented as follows:

[0144]

[0145] in, Here is the attention weight matrix. The Softmax activation function is used. For query vector, For key vectors, This is the transpose of the key vector. The dimension of the key vector. It is the square root function.

[0146] The fused output can be represented as:

[0147]

[0148] Where V is the value vector, and Output is the fused anomaly semantic representation.

[0149] Step 3.2: Merge features Input the anomaly classification header and risk scoring header, and output the anomaly categories respectively. and risk score :

[0150]

[0151]

[0152] in, As an anomaly category, For anomaly classifier, As a feature of fusion, As a risk score, As the risk score weight, It is a feedforward network. This biases the risk score. Furthermore, the overall risk score can also incorporate contextual factors. Perform weighted adjustment:

[0153]

[0154] in, The revised risk score, and These are the weighting coefficients. As a risk score, The context factor represents the scenario and is determined by the device type, task stage, historical anomaly frequency, and environmental complexity.

[0155] Step 4.1: As Figure 5 As shown, the output exception category and the revised risk score Then, based on the image data and text data, the abnormal location and inspection task context are obtained. The abnormal location includes one or more of the equipment number, component name, area name or image positioning area. The inspection task context includes one or more of the task work order information, equipment type, inspection round, historical abnormal records or current inspection stage.

[0156] Based on the abnormality categories identified during park inspections, suggested actions are pre-defined. These actions include one or more of the following: verification, recording, shutdown for inspection, power outage, component replacement, or reporting for maintenance.

[0157] Based on the anomaly category, risk score, inspection task context, anomaly location, and suggested actions, a set of feedback semantic slots is constructed. The formula is as follows:

[0158]

[0159] in, To provide a set of semantic slots for feedback, Functions are built to provide feedback semantic slots. As an anomaly category, As a risk score, This is an abnormal location. For suggested actions, For the context of the inspection task.

[0160] Step 4.2: Determine the feedback priority based on the risk score, as shown below:

[0161]

[0162] in, As a feedback priority, Determine the function for priority. Risk score;

[0163] Feedback priorities are divided into low, medium, and high priorities based on risk scores, as shown in the following expression:

[0164]

[0165] in, As a feedback priority, Low priority As a risk score, Medium priority As a high priority, The risk score threshold is one. The risk score threshold is two.

[0166] Step 4.3: Based on the set of feedback semantic slots and feedback priority, select the corresponding template from the preset feedback template set and generate candidate feedback text:

[0167]

[0168] in, For candidate feedback text, For template matching and filling functions, To provide a set of semantic slots for feedback, Prioritize feedback;

[0169] The candidate feedback text is processed by terminology normalization and redundancy reduction to obtain the final feedback text:

[0170]

[0171] in, For the final feedback text, For text normalization functions, For candidate feedback text;

[0172] For example, when the system detects that "the pressure gauge reading of pump No. 2 is abnormal and accompanied by high-frequency abnormal noise", the generated feedback text can be "Pump No. 2 pressure is abnormal and accompanied by abnormal noise. It is recommended to immediately review and prepare to stop the machine for inspection".

[0173] Step 4.4: Generate voice control parameters based on the final feedback text, feedback priority, and environmental noise parameters:

[0174]

[0175] in, Here, N represents the voice control parameter, and N represents the ambient noise parameter. For the final feedback text, As a feedback priority, Function for generating voice control parameters;

[0176] Voice control parameter set include

[0177]

[0178] in, For voice control parameter set, For speech rate parameters, For volume parameters, For pitch parameters, For pause parameters;

[0179] After prosodic annotation of the final feedback text, it is input into the speech synthesis module to generate a speech feedback signal:

[0180]

[0181]

[0182] in, This represents the text to be synthesized with prosodic annotations. This represents the prosody annotation function. For the final feedback text, Here is the set of voice control parameters, and F is the voice feedback signal. This is a speech synthesis function.

[0183] Specifically, the abnormality categories include one or more of the following: looseness, leakage, abnormal noise, overheating, abnormal readings, damage to appearance, or alarm abnormalities; environmental noise parameters can be represented by sound pressure level, noise energy, or noise level calculated from the on-site audio signal.

[0184] Step 4.5: The display module synchronously outputs graphical prompts, including anomaly boxes, risk colors, device numbers, and suggested action buttons, to achieve collaborative feedback between voice and graphics.

[0185] Step 5: Dynamically adjust the feedback results based on the inspection scenario and task stage, including:

[0186] When the noise in the inspection area exceeds the set noise threshold, the weights of the visual and OCR results are increased by the set weight step size.

[0187] When the screen is obstructed or the light intensity is lower than the set light intensity threshold, the constraint weights of audio and task text are increased by the set weight step size.

[0188] When a high-risk anomaly is detected, high-priority voice and visual cues will be output first.

[0189] Once the inspection personnel confirm the anomaly or complete the review, they will update the system's anomaly status and prompting policy.

[0190] Step 6: To better adapt the model to the park equipment inspection scenario, training is performed by freezing the bottom general feature layer and fine-tuning the higher task-related layers. The parameter update formula during fine-tuning is expressed as:

[0191]

[0192] in, These are the current model parameters. For learning rate, For the total loss function, Total loss function For parameters The gradient. In this embodiment, the total loss function consists of the anomaly classification loss, text generation loss, and cross-modal alignment loss:

[0193]

[0194] in, For the total loss function, This represents the anomaly classification loss. This indicates the loss in feedback text generation. This represents the alignment loss between visual, audio, and text features. The weighting coefficient is one. The weighting coefficient is two. The weighting factor is three. Furthermore, edge deployment performance can be optimized through distillation and quantization.

[0195] Example 2

[0196] Based on the same inventive concept as Embodiment 1, this embodiment introduces a park equipment inspection anomaly identification and intelligent interaction device, including:

[0197] The environmental perception module is used to acquire environmental perception data of the park, which includes image data, audio data, and text data.

[0198] The inspection feedback module is used to input environmental perception data into the intelligent inspection model of park equipment, and the intelligent inspection model of park equipment outputs feedback results.

[0199] The execution steps of the intelligent equipment inspection model in the park are as follows:

[0200] Preprocess the image data, audio data, and text data separately;

[0201] The processed image, audio, and text data are input into the cross-modal fusion module, which outputs anomaly categories and risk scores.

[0202] Intelligent feedback results are generated based on anomaly category and risk score;

[0203] The feedback results are dynamically adjusted based on the inspection scenario and task stage.

[0204] Example 3

[0205] This embodiment introduces a system for identifying and intelligently interacting with abnormal equipment inspections in a park, such as... Figure 6 As shown, the camera is used to capture images of the equipment to be inspected and its surrounding environment. The resolution is preferably set to 1080×1080 or 1280×720 to ensure clear details and meet the requirements of edge computing. The microphone is used to collect the operating sound of the equipment and the voice input of the inspectors. The sampling rate is preferably 16kHz or 48kHz. The computing processing unit is used to run visual coding, speech recognition, text coding and cross-modal fusion models. The audio output device includes a speaker or headphones for broadcasting inspection anomaly reminders and handling suggestions. The display device is used to display the anomaly selection results, equipment number, risk level and suggested actions.

[0206] Example 4

[0207] This embodiment describes the intelligent feedback process for scenarios and tasks. When inspection personnel perform routine inspections in the pump room area, the system first acquires images of the pump body, valves, and pressure gauges through a camera, collects equipment operating sounds through a microphone, and reads the task text "Check the pressure and valve status of pump No. 2" from the task list. When the system detects that the pressure gauge reading exceeds the limit, there are damp and reflective marks in the valve area, accompanied by high-frequency abnormal friction sounds, the cross-modal fusion module outputs a combined abnormal result of "pressure abnormality + suspected leakage + abnormal noise". The system prioritizes generating voice and graphic feedback based on risk scores, stating "Suspected leakage in the valve area of ​​pump No. 2, abnormal pressure, please check and record first," thereby helping inspection personnel quickly locate and handle the abnormality.

[0208] Example 5

[0209] This embodiment introduces experiments and performance optimizations, comparing the proposed solution with a single-vision detection solution in three typical scenarios: computer room inspection, pump room inspection, and power distribution equipment inspection. The experimental procedure includes: acquiring inspection images and audio, performing voice interrogation, conducting multimodal fusion recognition, outputting feedback results, and recording manual review conclusions. The experiments show that the proposed solution outperforms single-vision-based solutions in terms of complex anomaly recognition, false alarm control, and feedback executability. After lightweight distillation of the model and fine-tuning for the task scenarios, the inference latency of the system on the edge processing unit is further reduced, making it more suitable for real-time field applications.

[0210] In summary, through the above embodiments, the present invention achieves anomaly identification and intelligent interaction for park equipment inspection scenarios by using a complete technical chain of image processing, speech recognition, text encoding, cross-modal fusion, anomaly scoring, intelligent feedback, and fine-tuning optimization.

[0211] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0212] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0213] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0214] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0215] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for identifying and intelligently interacting with abnormal equipment inspections in a park, characterized in that, include: Acquire environmental perception data of the park, including image data, audio data, and text data; The environmental perception data is input into the intelligent inspection model of park equipment, and the intelligent inspection model of park equipment outputs feedback results. The intelligent inspection model for park equipment executes the following steps: The image data, audio data, and text data are preprocessed respectively; The processed image, audio, and text data are input into the cross-modal fusion module, which outputs anomaly categories and risk scores. Intelligent feedback results are generated based on the anomaly category and risk score; The feedback results are dynamically adjusted based on the inspection scenario and task stage.

2. The method for identifying and intelligently interacting with abnormal equipment inspections in a park according to claim 1, characterized in that, The preprocessing includes: After performing denoising, white balance correction, brightness normalization, resolution adjustment and target region cropping on the image data, an image input suitable for encoder processing is obtained. The image input is then converted into visual features by a visual encoder. After performing endpoint detection, noise suppression, gain control, and short-time spectrum transformation on the audio data, audio features are extracted. After performing word segmentation, removal of redundant stop words, key entity recognition, and sequence concatenation on the text data, the user's voice input recognition model is converted into speech text. The speech text, along with the OCR text and task text in the text data, are constructed into a unified text sequence, which is then input into the text encoder to form text features.

3. The method for identifying and intelligently interacting with abnormal equipment inspections in a park according to claim 2, characterized in that, The cross-modal fusion module performs the following steps: Perform dimensional unification and semantic alignment on the aforementioned visual features, audio features, and text features; Based on the aligned features, calculate the cross-modal attention weights and generate the fused output, as shown in the following formula: in, Here is the attention weight matrix. The Softmax activation function is used. For query vector, For key vectors, This is the transpose of the key vector. The dimension of the key vector. It is a square root function. For attention output, It is a value vector; The outputs of the cross-modal attention layers are integrated into a unified fusion feature. The formula is as follows: in, As a feature of fusion, For feature fusion function, As a visual feature, For audio features, Text features; The fusion features Input the anomaly classification header and risk scoring header, and output the anomaly categories respectively. and risk score The formula is as follows: in, As an anomaly category, For anomaly classifier, As a feature of fusion, As a risk score, As the weight for risk scoring, It is a feedforward network. This is for risk scoring bias.

4. The method for identifying and intelligently interacting with abnormal equipment inspections in a park according to claim 1, characterized in that, The image data prioritizes locating the dashboard area, valve area, wiring area, nameplate area, and warning sign area; The audio data is used to prioritize the identification of key acoustic events, including abnormal motor noises, leakage sounds, friction sounds, and alarm beeps. The text data is extracted first from the speech transcription results, OCR recognition results, and task work orders, including the equipment name, inspection location, abnormal keywords, and handling action keywords.

5. The method for identifying and intelligently interacting with abnormal equipment inspections in a park according to claim 1, characterized in that, The intelligent feedback result includes at least one of the following: abnormal object, abnormal location, abnormality level, and suggested action.

6. The method for identifying and intelligently interacting with abnormal equipment inspections in a park according to claim 3, characterized in that, The step of generating intelligent feedback results based on the anomaly category and risk score includes: The abnormal location and inspection task context are obtained based on the image data and text data. The abnormal location includes one or more of the following: equipment number, component name, area name, or image positioning area. The inspection task context includes one or more of the following: task work order information, equipment type, inspection round, historical abnormal records, or current inspection stage. Based on the abnormality categories of the park inspection, the suggested actions are preset, including one or more of the following: review, record, shutdown inspection, power outage, component replacement, or reporting for maintenance. Based on the anomaly category, risk score, inspection task context, anomaly location, and suggested action, a set of feedback semantic slots is constructed. The formula is as follows: in, To provide a set of semantic slots for feedback, Functions are built to provide feedback semantic slots. As an anomaly category, As a risk score, This is an abnormal location. For suggested actions, For the context of the inspection task; The feedback priority is determined based on the risk score, as follows: in, As a feedback priority, Determine the function for priority. Risk score; Based on the set of feedback semantic slots and feedback priority, a corresponding template is selected from the preset feedback template set and candidate feedback text is generated: in, For candidate feedback text, For template matching and filling functions, To provide a set of semantic slots for feedback, Prioritize feedback; The candidate feedback text is subjected to terminology normalization and redundancy reduction processing to obtain the final feedback text: in, For the final feedback text, For text normalization functions, For candidate feedback text; Voice control parameters are generated based on the final feedback text, feedback priority, and environmental noise parameters. in, Here, N represents the voice control parameter, and N represents the environmental noise parameter. For the final feedback text, As a feedback priority, Function for generating voice control parameters; The set of voice control parameters include in, For voice control parameter set, For speech rate parameters, For volume parameters, For pitch parameters, For pause parameters; After prosodic annotation of the final feedback text, it is input into the speech synthesis module to generate a speech feedback signal: in, This represents the text to be synthesized with prosodic annotations. This represents the prosody annotation function. For the final feedback text, Here is the set of voice control parameters, and F is the voice feedback signal. This is a speech synthesis function; Synchronously output graphical prompts, including anomaly boxes, risk colors, device numbers, and suggested action buttons.

7. The method for identifying and intelligently interacting with abnormal equipment inspections in a park according to claim 6, characterized in that, The feedback priority is divided into low priority, medium priority, and high priority based on the risk score, as expressed below: in, As a feedback priority, Low priority As a risk score, Medium priority As a high priority, The risk score threshold is one. The risk score threshold is two.

8. The method for identifying and intelligently interacting with abnormal equipment inspections in a park according to claim 1, characterized in that, The dynamic adjustment of feedback results based on the inspection scenario and task stage includes: When the noise in the inspection area exceeds the set noise threshold, the weights of the visual and OCR results are increased by the set weight step size. When the screen is obstructed or the light intensity is lower than the set light intensity threshold, the constraint weights of audio and task text are increased by the set weight step size. When a high-risk anomaly is detected, high-priority voice and visual cues will be output first. Once the inspection personnel confirm the anomaly or complete the review, they will update the system's anomaly status and prompting policy.

9. The method for identifying and intelligently interacting with abnormal equipment inspections in a park according to claim 1, characterized in that, Also includes: Based on historical inspection data, manual annotation results, and user operation feedback, the parameters of the intelligent inspection model for park equipment are fine-tuned and optimized online. The parameter update formula for fine-tuning is: in, These are the current model parameters. For learning rate, Total loss function For parameters gradient, This is the total loss function; The total loss function is composed of anomaly classification loss, text generation loss, and cross-modal alignment loss: in, For the total loss function, This represents the anomaly classification loss. This indicates the loss in feedback text generation. This represents the alignment loss between visual, audio, and text features. The weighting coefficient is one. The weighting coefficient is two. The weighting coefficient is three.

10. A device for identifying and intelligently interacting with abnormal equipment inspections in a park, characterized in that, include: The environmental perception module is used to acquire environmental perception data of the park, including image data, audio data, and text data. The inspection feedback module is used to input the environmental perception data into the intelligent inspection model of park equipment, and the intelligent inspection model of park equipment outputs feedback results. The intelligent inspection model for park equipment executes the following steps: The image data, audio data, and text data are preprocessed respectively; The processed image, audio, and text data are input into the cross-modal fusion module, which outputs anomaly categories and risk scores. Intelligent feedback results are generated based on the anomaly category and risk score; The feedback results are dynamically adjusted based on the inspection scenario and task stage.

Images