Intelligent inspection vehicle based on visual voice collaborative perception in mechanical workshop

Abstract

The application belongs to the technical field of intelligent inspection of mechanical workshops, and discloses an intelligent inspection vehicle for mechanical workshops based on visual-voice cooperative sensing, which comprises a mobile chassis, a work station positioning module, an equipment visual anchoring module, a workshop voice sensing module, a sound-visual reference correlation module, an equipment anomaly review module and an inspection event closed loop module; a visual inspection area is established through equipment work station identification and visual collection reliability is calculated, staff voice and voice segment reliability are obtained according to equipment noise baseline, a target equipment is determined in combination with sound source direction, staff pointing direction and voice semantics, a review pose is re-collected to generate visual evidence when shielding and sound-visual results are inconsistent, and an inspection event is generated by fusing sound-visual anomalies and review results; the application can reduce abnormal misjudgment caused by shielding, noise and ambiguous voice reference, and improve the accuracy and on-site applicability of mechanical workshop equipment inspection.

Description

Technical Field

[0001] This invention belongs to the field of intelligent inspection technology in mechanical workshops, and particularly relates to an intelligent inspection vehicle for mechanical workshops based on visual and voice collaborative perception. Background Technology

[0002] Machining workshops typically house machining centers, CNC lathes, and auxiliary production equipment. During equipment operation, periodic inspections of status lights, control doors, equipment bottom leaks, chip discharge ports, and instrument status are required. Meanwhile, on-site personnel can detect anomalies based on equipment operating sounds and actual operating conditions. Therefore, combining the visual inspection capabilities of inspection vehicles with on-site voice feedback from personnel can improve the efficiency of equipment anomaly detection and location. This approach is suitable for machining workshops with a large number of equipment, fixed inspection channels, and a significant workload for manual inspections.

[0003] (1) Existing intelligent inspection vehicles mainly rely on cameras to identify the surface condition of equipment from a single perspective. However, in the machine workshop, there are often tool carts, material boxes, workpieces to be processed and staff members that block the local area of ​​the equipment. At the same time, oil reflection, changes in the observation angle and the parking deviation of the inspection vehicle will also affect the image judgment. The existing method lacks the judgment of the effectiveness of visual acquisition and the verification process of actively changing the observation position according to the geometric relationship of the blockage, which is prone to missed detection and false detection.

[0004] (2) The main spindle, motor, cutting fluid pump, chip conveyor and compressed air equipment in the machine shop generate continuous noise and transient impact noise. When the staff use "this", "here" and "bottom right corner" to make on-site voice feedback, the existing inspection equipment usually cannot accurately match the voice content with the specific equipment and specific parts in the camera screen. Visual abnormalities and voice abnormalities are prone to cross-equipment mismatch, making it difficult to form a complete inspection event with equipment identity, abnormal parts and verification evidence. Summary of the Invention

[0005] This invention aims to solve the problem that existing mechanical workshop inspection equipment is easily affected by a single observation perspective under conditions of machine tool operating noise, compressed air impact noise, tool cart obstruction, material frame obstruction, and dense arrangement of multiple devices. When workers use "this machine" or "here" for voice feedback, it is difficult to determine the actual device being referred to, leading to mismatch between visual and voice abnormalities. At the same time, it also solves the problem that existing inspection methods still rely on the original observation position for repeated identification when the initial visual result is unreliable, and cannot actively remove obstructions to obtain effective verification evidence.

[0006] The technical solution adopted in this invention is as follows: it includes a mobile chassis, a workstation positioning module, an equipment visual anchoring module, a workshop voice perception module, a voice and visual reference association module, an equipment anomaly verification module, and an inspection event closed-loop module.

[0007] The mobile chassis is equipped with a laser ranging unit, drive wheels, encoders, and anti-collision contact edges. It moves between different equipment stations according to the mechanical workshop aisle map, and forms a visual acquisition pose based on the position of the equipment operating side after entering the equipment station.

[0008] The workstation positioning module obtains the equipment workstation identification plate, determines the identity of the currently inspected equipment and the equipment workstation location based on the equipment number and spatial location in the equipment workstation identification plate, generates the equipment workstation positioning result, and sends the equipment workstation positioning result to the equipment visual anchoring module.

[0009] The equipment visual anchoring module receives the equipment station positioning results, establishes the current equipment station coordinate system based on the actual size and image position of the equipment station identification sign, reads the standard inspection template corresponding to the equipment number, and maps the three-color status light area, the operating door closing area, the equipment bottom leakage area, the chip discharge port area, and the instrument display area to the current acquisition screen to generate the visual inspection area.

[0010] The equipment visual anchoring module generates visual acquisition reliability based on the visibility ratio of the target visual inspection area, the degree of image motion blur, the equipment identification status, and the position mapping error of the standard inspection template. Based on the visual acquisition reliability, the target visual inspection area is divided into valid visual samples and visual samples to be verified.

[0011] The equipment visual anchoring module generates visual anomaly results by combining the actual illumination status of the status lights, the actual gap of the operating door, the actual area of ​​the liquid patches at the bottom of the equipment, the actual height of the chip accumulation, and the status of the instruments with visual changes at different observation times.

[0012] The workshop voice perception module is equipped with a microphone array. After entering different equipment workstations, it collects the background sound of the current equipment operation, establishes the equipment noise baseline corresponding to the current equipment workstation, distinguishes continuous equipment noise, impact noise and staff voice based on the equipment noise baseline, and obtains the staff voice text and sound source direction.

[0013] The workshop voice perception module generates voice segment reliability based on the proportion of impact noise frames in the worker's voice segment, the relationship between the effective frequency band energy of the voice and the background noise energy of the equipment, and the duration of no effective voice. It also extracts equipment number, equipment location, abnormal action information, and spatial description information from the worker's voice text to generate voice perception results.

[0014] The audio-visual association module receives the visual inspection area, visual acquisition reliability, voice perception results, and voice segment reliability. It maps the sound source direction obtained by the microphone array to the camera observation coordinate system and generates the worker's pointing direction based on the position of the shoulder, elbow, and wrist when the worker's upper limbs are within the effective range of the image.

[0015] The audio-visual referencing and association module generates audio-visual referencing and association values ​​based on the differences between the direction of the staff's voice source and the visual orientation of the candidate equipment, the positional relationship between the staff's pointing direction and the candidate visual inspection area, the semantic matching degree between the voice text and the equipment number and equipment part, and in combination with the reliability of visual acquisition and the reliability of voice segments.

[0016] The audio-visual reference association module determines the target device and target visual inspection area that the staff's voice actually corresponds to based on the audio-visual reference association value corresponding to each candidate device. When the difference between the highest audio-visual reference association value and the second highest audio-visual reference association value is insufficient to determine a unique target, a reconfirmation request is generated to prevent voice reference errors between adjacent devices.

[0017] The equipment anomaly verification module receives the target equipment association results, visual anomaly results, and visual acquisition reliability. When the semantics of the staff’s voice anomaly are inconsistent with the visual anomaly results, or when the target visual inspection area is obstructed, a lateral verification displacement is generated based on the lateral width of the obstruction, the distance from the obstruction to the camera, the distance from the target visual inspection area to the camera, and the available lateral space of the inspection channel.

[0018] The mobile chassis changes the observation position according to the lateral verification displacement, re-identifies the equipment workstation sign and re-captures the target visual inspection area. When the current equipment identity can still be confirmed at the verification position, the verification image is associated with the original inspection event to generate the equipment anomaly verification result.

[0019] When the calculated lateral verification displacement exceeds the allowable movement range of the inspection channel, the equipment abnormality verification module restricts the mobile chassis from moving to the maximum allowable verification position and controls the camera to change the viewing angle, thereby improving the visibility of the target visual inspection area while maintaining the chassis's safe passage.

[0020] For the leakage area at the bottom of the equipment, the equipment anomaly verification module compares the changes in the boundary of the liquid patch, the changes in the actual area of ​​the patch, and the spatial relationship between the patch and the liquid outlet position at the bottom of the equipment at multiple consecutive observation times, and distinguishes the ground reflection area that moves with the change of observation angle from the continuously expanding liquid patch.

[0021] The inspection event closed-loop module receives visual anomaly results, visual acquisition reliability, voice anomaly semantics, voice segment reliability, audio-visual referential correlation value, and equipment anomaly review results. Based on visual anomaly evidence, voice anomaly evidence, audio-visual object consistency, anomaly persistence before and after review, and equipment safety level, it generates inspection event confidence values.

[0022] When visual and voice anomalies point to the same device or the same part of the device, the inspection event closed-loop module increases the contribution of audio-visual collaborative evidence. When visual and voice anomalies point to different devices or different parts of the device, no audio-visual collaborative enhancement results are generated, thereby avoiding the incorrect fusion of anomaly information from different devices.

[0023] The inspection event closed-loop module generates normal inspection records, re-verification events, and equipment abnormality events based on the inspection event confidence value. The equipment abnormality event is associated and saved with the equipment number, abnormal location, abnormality type, on-site image, staff voice and text, inspection time, and equipment abnormality verification result.

[0024] The beneficial effects of this invention are as follows: (1) Establish the current equipment station coordinate system through the equipment station identification sign, and generate the visual acquisition reliability based on the visible ratio of the target visual inspection area, the degree of image motion blur and template mapping error, so that the anomaly identification is based on the effective visual acquisition conditions, and reduces the visual false alarms caused by squinting, occlusion and ground reflection in the mechanical workshop.

[0025] (2) By generating audio-visual referential association results through the direction of the staff’s voice source, the direction the staff points, and the semantics of speech, the abnormal voice information corresponding to “this machine” and “here” is bound to the specific equipment and the specific inspection location, thus solving the cross-equipment mismatch problem caused by determining the voice object only according to the distance under the condition of dense equipment layout.

[0026] (3) When the reliability of visual results is insufficient and there is a conflict between visual and speech results, a verification pose is generated based on the occlusion geometry and the remaining space of the inspection channel. Abnormal evidence is obtained again after the observation position is changed by moving the chassis. The confidence value of the inspection event is generated in combination with the verification continuous state to solve the problem that the abnormality cannot be confirmed even after repeated identification when the original observation position is occluded. Attached Figure Description

[0027] The accompanying drawings are provided to further understand the present invention and form part of the specification. They are used together with the embodiments of the present invention to explain the invention and do not constitute a limitation thereof.

[0028] Figure 1 This is a schematic diagram of the overall structure and processing flow of the inspection vehicle proposed in this invention; Figure 2 This is a schematic diagram of the audio-visual collaborative perception and anomaly verification process of the present invention. Detailed Implementation

[0029] Example 1, see Figures 1-2This invention provides an intelligent inspection vehicle for a mechanical workshop based on visual and voice collaborative perception, including a mobile chassis, a workstation positioning module, an equipment visual anchoring module, a workshop voice perception module, an audio-visual association module, an equipment anomaly verification module, and an inspection event closed-loop module. The mobile chassis is equipped with a laser ranging unit, drive wheels, an encoder, and anti-collision contact edges. The equipment visual anchoring module includes a pan-tilt camera and an edge vision processing unit. The workshop voice perception module includes a four-channel microphone array, a voice processing unit, and a speaker. The audio-visual association module, the equipment anomaly verification module, and the inspection event closed-loop module are deployed on the vehicle-mounted edge computing unit.

[0030] This embodiment is applied to the precision machining workshop of a machinery manufacturing enterprise. The workshop is equipped with 24 vertical machining centers and 8 CNC lathes. The equipment is arranged in four rows, and an inspection channel with a width of 1.8m to 2.2m is formed between adjacent rows of equipment. During the workshop production, the machine tool spindles, cutting fluid pumps, chip conveyors and compressed air nozzles are constantly working. The sound pressure level in the inspection channel is between 78dB and 92dB. There are intermittent obstructions caused by tool carts, material frames and workpieces to be processed in front of the equipment.

[0031] Each piece of equipment under inspection has an equipment workstation identification plate on the operating side. The equipment workstation identification plate records the equipment number and has a fixed external dimension. After the equipment visual anchoring module recognizes the equipment workstation identification plate, it establishes the workstation coordinate system of the current equipment and retrieves the corresponding visual inspection area according to the equipment model. The visual inspection area includes the three-color status light area, the operating door closing area, the equipment bottom leakage area, the chip discharge port area, and the instrument display area.

[0032] The inspection vehicle enters each equipment station sequentially according to the pre-established workshop passage map. It slows down and performs visual anchoring at a distance of 1.2m to 1.8m from the equipment operating side. After successful visual anchoring, the pan-tilt camera captures close-up images of the status light area, the operating door closing area, the equipment bottom leakage area, the chip discharge port area, and the instrument display area. At the same time, the workshop voice perception module continuously detects the voice input of the staff around the inspection vehicle.

[0033] Staff can directly say phrases containing equipment numbers, such as "the lower right corner of machine tool number 12 is leaking oil" or "machine number 8 just alarmed." They can also use descriptive phrases like "this spindle is making a strange sound," "there is oil here," or "this door is not closed properly" when they are near the target machine tool. The audio-visual descriptive association module simultaneously reads the direction of the sound source from the microphone array, the direction of the staff's upper limbs detected by the camera, and the candidate area of ​​the equipment workstation in the current image for phrases without explicit equipment numbers, thereby determining the target equipment corresponding to the voice description.

[0034] The equipment anomaly verification module does not directly generate fault events based on the results of a single image recognition. When the target visual inspection area is obstructed by a tool cart, material frame, or worker's body, or when the voice description is inconsistent with the initial visual recognition result, the equipment anomaly verification module generates a verification pose based on the position of the obstruction, the position of the target area, and the remaining space in the inspection channel. It then controls the mobile chassis to change the observation position laterally and re-acquire the target visual inspection area.

[0035] The inspection event closed-loop module integrates visual anomaly results, voice anomaly semantics, audio-visual referential association results, and the anomaly persistence status before and after review to generate inspection event confidence values. Based on the inspection event confidence values, the current results are divided into normal inspection records, re-review events, and equipment anomaly events. Equipment anomaly events save equipment number, anomaly location, anomaly type, on-site images, staff voice text, inspection time, and review results.

[0036] This embodiment forms a continuous processing chain consisting of "visual anchoring of equipment workstations - credibility of workshop noise voice - audio-visual reference area - occlusion removal verification - closed-loop confirmation of abnormal evidence". This makes voice no longer an independent text annotation, and visual recognition no longer independently determine the fault result. This solves the problems of mismatched voice reference objects and false alarms of single-view images under high noise, dense equipment and temporary occlusion conditions in mechanical workshops.

[0037] Example 2: Based on all the above examples, the device visual anchoring module specifically performs device identification, visual inspection area positioning, and visual acquisition reliability assessment.

[0038] After the inspection vehicle arrives at the equipment station, it first reads the equipment station identification sign. The equipment station identification sign has a known width and height. The edge vision processing unit completes perspective correction based on the position of the four corners of the equipment station identification sign in the image, and establishes the size conversion relationship of the current image based on the actual size of the equipment station identification sign, so that the area of ​​the leakage area, the gap of the operating door and the height of the chip accumulation can be converted from the image pixel quantity to the actual size quantity.

[0039] The equipment visual anchoring module reads the standard inspection template of the corresponding equipment according to the equipment number. The standard inspection template saves the relative positional relationship between the equipment workstation identification plate and the three-color status light area, the operating door closing area, the equipment bottom leakage area, the chip discharge port area and the instrument display area. After the equipment workstation identification plate is positioned, the standard inspection template is mapped to the current acquisition screen.

[0040] In the machine shop, tool carts and material boxes are often temporarily parked in front of the equipment. Therefore, the equipment visual anchoring module does not directly recognize the target area entering the screen as a valid acquisition. Instead, it further calculates the visible ratio of the target area, the degree of image motion blur, and the template mapping position error to generate the visual acquisition reliability.

[0041] Calculate the visual acquisition reliability of the current visual inspection area of ​​the i-th equipment station according to Formula 1:

[0042] In Formula 1, This represents the reliability of visual data acquisition at the i-th equipment station. This indicates the confidence level of identification of the equipment workstation identification sign. This indicates the proportion of the unobstructed portion of the target visual inspection area to the total area of ​​the complete target visual inspection area. This represents the amount of motion blur obtained based on the degree of edge diffusion in a continuous image. This indicates the positional mapping error between the current equipment workstation identification sign and the standard inspection template after perspective correction. This indicates the upper limit of the allowed location mapping error.

[0043] When the visual acquisition reliability reaches 0.70, the device visual anchoring module marks the current target visual inspection area as a valid visual sample. When the visual acquisition reliability is between 0.45 and 0.70, the current target visual inspection area is marked as a visual sample to be reviewed. When the visual acquisition reliability is below 0.45, the device does not use the current image to directly generate a device anomaly result and sends an occlusion review request to the device anomaly review module.

[0044] The three-color status light area is first divided according to the equipment standard template, and then the actual lighting status of the red, yellow and green lights is compared with the normal light status corresponding to the current operating stage of the equipment. The closing gap of the operating door area is obtained according to the actual distance between the door frame baseline and the edge of the door. The leakage area at the bottom of the equipment is calculated by calculating the actual area of ​​the continuous oil patch through the ground area after perspective correction. The chip discharge port area is calculated according to the chip accumulation height based on the size baseline of the equipment base. The instrument display area reads the pressure gauge pointer position and the digital instrument display status.

[0045] For bottom leakage detection of equipment, highly reflective cutting fluid is not directly judged as oil leakage based on its color. Instead, ground images are continuously collected at three observation times, and visual abnormality evidence is formed based on the spatial relationship between the continuous expansion of the patch boundary, the change of surface highlights, and the location of the liquid outlet at the bottom of the equipment. Reflective areas that appear briefly and whose location changes with personnel movement are not recorded as equipment abnormality events.

[0046] Regarding parameter adjustments: In processing areas with heavy oil mist, increase the limit corresponding to motion blur. When there is oil on the equipment workstation sign, reduce the single determining role of identity recognition confidence and increase the limit of template mapping position error. When the tool cart on the front of the equipment appears frequently, keep the lower limit of the visual samples to be reviewed at 0.45 to avoid the occluded areas being directly discarded.

[0047] Through the above processing, the equipment visual anchoring module first determines whether the current viewing angle has effective inspection conditions before visual anomaly identification, and uses the equipment workstation identification sign to complete the actual size conversion, solving the problem of visual anomaly size distortion caused by squinting, reflection and temporary obstruction in the machine shop.

[0048] Example 3: Based on all the above examples, the workshop voice perception module specifically performs workshop noise segmentation, worker voice extraction, and voice segment reliability assessment.

[0049] The four-channel microphone array is installed on the front of the inspection vehicle. The distance between the center of the microphone array and the optical axis of the pan-tilt camera in the horizontal direction is 85mm. The vehicle-mounted edge computing unit pre-records the installation angle of the microphone array relative to the coordinate system of the inspection vehicle, so that the direction of the sound source can be converted to the camera's observation coordinate system.

[0050] After the inspection vehicle enters each equipment station, the workshop voice perception module first collects the background sound of the equipment without personnel speaking for 2 seconds, extracts the continuous background spectrum formed by the stable rotation of the spindle, the operation of the cutting fluid pump and the operation of the chip conveyor, and establishes the noise baseline of the current equipment station. Subsequent voice detection does not directly use the uniform noise template of the entire workshop.

[0051] Compressed air jets, workpiece collisions, and cutting tools falling into the material frame create impact noise with short durations. The workshop voice perception module divides the voice analysis frames into 20ms intervals, marks the analysis frames with sudden energy increases but lacking continuous voice formant structures as impact noise frames, and records the number of impact noise frames before voice recognition, instead of simply increasing the voice detection threshold to block all low-amplitude voice.

[0052] During the staff's speech, the microphone array determines the direction of the sound source based on the arrival time difference of the four-channel signals. At the same time, it extracts the equipment number, equipment location, abnormal action words, and degree words from the speech recognition results. The equipment number includes the specific equipment entity corresponding to "machine number 12" and "M12 machine tool". The equipment location includes "spindle", "operating door", "lower right corner", and "chip outlet". The abnormal action words include "oil leak", "alarm", "stuck", and "loose". The degree words include "just now", "always", "suddenly", and "serious".

[0053] The reliability of the j-th worker's speech segment is calculated using Formula 2:

[0054] In Formula 2, This represents the reliability of the speech segment of the j-th worker. This indicates the number of frames in a speech segment that are identified as impulse noise. This indicates the total number of analysis frames contained in the speech segment. This represents the ratio between the effective speech frequency band energy obtained after separating from the noise baseline of the current equipment station and the background equipment noise energy. This represents the reference energy ratio required to achieve stable speech recognition. This indicates the absence of effective speech duration due to impulse noise within a speech segment. Indicates the duration of the entire speech segment.

[0055] When the reliability of a voice segment is below 0.40, the workshop voice perception module does not use the voice segment to directly participate in the confirmation of equipment anomalies, but retains the sound source direction and timestamp. When the reliability of a voice segment is between 0.40 and 0.65, the voice segment is marked as a voice segment to be confirmed. When the reliability of a voice segment reaches 0.65, the identified text, sound source direction and abnormal semantics are sent to the audio-visual reference association module.

[0056] When the staff explicitly states the equipment number, the audio-visual reference association module first uses the equipment number to limit the range of candidate equipment. When the staff only uses "this one", "here", or "this location" to describe it, the audio-visual reference association module enters the reference area process and does not directly regard the machine tool closest to the inspection vehicle as the voice target.

[0057] Regarding parameter adjustments: the reference energy ratio is increased in the continuous cutting station, the impact noise frame is increased in the intermittent stamping area to suppress the reliability of voice segments, and the threshold for voice segments to enter the pending confirmation state is increased when the inspection vehicle is more than 3m away from the speaker, thereby reducing the probability of long-distance speech interfering with the inspection of adjacent equipment.

[0058] Through the above processing, the workshop voice perception module re-establishes the voice reliability basis for the on-site noise of each equipment station, so that continuous machine tool noise and transient impact noise are processed separately, solving the problems that uniform noise reduction parameters can easily identify impact noise as speech and excessively eliminate weak speech.

[0059] Example 4: Based on all the above examples, the audio-visual referencing and association module specifically performs sound source direction mapping, worker pointing direction extraction, and semantic device candidate matching.

[0060] The equipment visual anchoring module simultaneously detects the equipment workstation area and the speaking staff in the camera frame. When the speaking staff's upper limbs are within the effective range of the frame, the positions of the shoulder, elbow, and wrist are extracted. Based on the direction from the elbow to the wrist, a pointing ray of the staff is generated, and the shortest distance between the pointing ray of the staff and the center of each equipment visual inspection area is calculated.

[0061] After the microphone array obtains the direction of the staff's voice, it transforms the direction of the voice to the camera coordinate system and calculates the angle difference between the direction of the voice and the visual azimuth angle formed by the center of each equipment station relative to the inspection vehicle, thereby eliminating equipment in the picture that is close to the staff but is obviously inconsistent with the direction of the speech.

[0062] When the voice text contains a clear device number, the device number consistency is set to the highest semantic matching state. When the voice text only contains device parts, the candidate device is checked for corresponding visual inspection areas according to the standard inspection template. When the voice text also contains spatial descriptions such as "bottom right corner", "door edge", and "in front of the chip discharge port", the spatial descriptions are matched with the visual inspection areas of the candidate devices.

[0063] Calculate the audiovisual referential association value between the j-th speech segment and the i-th equipment station according to Formula 3:

[0064] In Formula 3, This represents the audiovisual referential association value between the j-th speech segment and the i-th equipment station. This represents the reliability of visual data acquisition at the i-th equipment station. This represents the reliability of the j-th speech segment. This indicates the direction angle of the sound source obtained by the microphone array from the staff. This represents the visual azimuth angle of the center of the i-th device station in the camera coordinate system. This indicates the maximum permissible acoustic-visual azimuth difference. This represents the shortest distance between the ray pointed to by the staff and the center of the visual inspection area of ​​the i-th device. Indicates the allowed matching distance. This represents the semantic matching value between the device number, device location, and spatial description in the voice text and the standard inspection template for the i-th device workstation. These represent the correlation coefficients corresponding to the sound source location, the staff's direction, and the semantic matching, respectively.

[0065] When the staff member's upper limbs are not in the camera's field of view, the correlation coefficient corresponding to the staff member's pointing is temporarily assigned to the sound source location and semantic matching. When the staff member clearly states the device number, the correlation coefficient corresponding to the semantic matching is increased. When the staff member uses pronouns such as "this one" or "here" and the upper limbs are fully visible, the correlation coefficient corresponding to the staff member's pointing is increased.

[0066] When the difference between the highest audio-visual reference correlation value and the second highest audio-visual reference correlation value is less than 0.12, the audio-visual reference correlation module does not force the selection of the target device. The inspection vehicle outputs a confirmation prompt "Please point to the device and explain again" through the speaker, and recalculates the audio-visual reference correlation value using the next voice segment and the new staff pointing result.

[0067] When the highest audio-visual association value reaches 0.70 and the difference between it and the second highest audio-visual association value reaches 0.12, the corresponding equipment station is identified as the voice target, and the voice anomaly semantics is bound to the equipment station number and the specific visual inspection area.

[0068] For example, when a worker stands between machining centers 12 and 13 and says, "There is oil in the lower right corner of this machine," the target equipment cannot be reliably determined simply by distance. In this embodiment, the direction of the worker's voice source is used to limit the candidate direction of the equipment, the direction of the arm pointing to the ray is used to determine the target equipment, and finally the semantic positional consistency between "lower right corner" and the leakage area at the bottom of the equipment is used to complete the binding.

[0069] Regarding parameter adjustments: in workshops with small equipment spacing, reduce the maximum permissible acoustic-visual orientation difference; when staff frequently report abnormalities from the side of the equipment, increase the permissible distance between the staff's pointing ray and the center of the visual inspection area; and after the equipment number speech recognition is stable, increase the contribution of the equipment number to the semantic matching value.

[0070] Through the above processing, the referential information in the voice can be spatially bound to the specific equipment and inspection area currently observed by the camera, solving the problem that "this" and "here" cannot be determined as the actual referent when multiple pieces of equipment in the machine shop enter the frame at the same time.

[0071] Example 5: Based on all the above examples, the device anomaly verification module specifically performs visual anomaly extraction, audio-visual conflict judgment, and occlusion removal verification pose generation.

[0072] For the three-color status light area, the edge vision processing unit extracts the actual lit light position and compares it with the equipment operation status light position recorded in the standard inspection template. For the operating door closing area, the actual gap between the door edge and the door frame baseline is calculated using the size conversion relationship formed by the equipment workstation identification plate. For the equipment bottom leakage area, the actual area of ​​the continuous liquid patch and the area growth at three consecutive observation times are calculated. For the chip discharge port area, the actual height of the highest point of the chip profile relative to the chip discharge port reference surface is calculated.

[0073] Input the abnormality levels of status lights, operating door gaps, oil patch expansion, chip accumulation, and instrument boundary violations into the workstation visual anomaly discrimination unit to generate a visual anomaly value V_i. The larger the visual anomaly value, the higher the degree to which the visual evidence of the current equipment workstation deviates from the normal operating conditions. The visual anomaly value only directly participates in the anomaly event calculation when the corresponding visual inspection area meets the valid visual sample conditions.

[0074] The semantic analysis of speech anomalies generates the degree of speech anomaly based on terms such as "oil leak," "alarm," "stuck," "loose," and corresponding degree words. When the degree of voice abnormality is high and the value of visual abnormality is low, and the target visual inspection area belongs to the visual sample to be reviewed, the equipment abnormality review module will prioritize the existence of visual occlusion rather than the staff's report as wrong.

[0075] When a tool cart, material frame, or workpiece obstructs the target visual inspection area, the equipment anomaly verification module extracts the width of the obstruction, the distance from the obstruction to the current camera, and the distance from the target visual inspection area to the current camera from the depth measurement results, and determines the lateral verification movement direction based on the remaining space on both sides of the inspection channel.

[0076] Calculate the lateral verification displacement required to remove the current occlusion using Formula 4:

[0077] In Formula 4, This represents the lateral verification displacement generated for the i-th equipment station. This indicates the direction of movement, determined by the passable width on both sides of the inspection vehicle; a negative value is taken to the left and a positive value to the right. This indicates the available lateral movement distance in the corresponding direction after meeting the chassis safety margin. This represents the actual width of the obstruction in the lateral direction, and m_s represents the visual safety margin that the observed ray needs to retain when crossing the edge of the obstruction. This represents the longitudinal distance from the target visual inspection area to the current camera. This represents the longitudinal distance from the obstruction to the current camera, and satisfies... .

[0078] Formula 4 is derived from the geometric relationship of the observation ray from the target center to the plane of the obstruction after the camera moves. When the calculated theoretical lateral displacement is greater than the currently available lateral movement distance of the inspection channel, the equipment anomaly verification module does not perform movement beyond the channel boundary, but moves to the maximum allowable verification position and controls the pan-tilt unit to change the observation angle.

[0079] After the inspection vehicle arrives at the verification location, it reacquires the equipment station identification plate and recalculates the reliability of visual acquisition. The new image is merged with the original inspection event only when the equipment station number can still be confirmed at the verification location, in order to prevent the images of adjacent machine tools from being incorrectly written into the current equipment event after lateral movement.

[0080] When the area of ​​the leaking area at the bottom of the equipment increases significantly after verification and the abnormal voice message indicates oil leakage, the consistency of the current inspection event is improved. When the original abnormal patch disappears after verification and the patch location is consistent with the high-light area on the ground, the visual anomaly value is reduced and the original image is saved as a false positive sample for reflection.

[0081] Regarding parameter adjustments: When the inspection channel width is less than 1.6m, increase the visual safety margin and reduce the lateral movement speed. For workstations with a large number of tool carts, prioritize the use of mobile chassis for verification. When personnel are densely packed, stop lateral verification and wait for the channel to be restored to avoid changing the normal passage conditions for workers in order to obtain a more complete image.

[0082] Through the above processing, the inspection vehicle can proactively change its observation posture in response to conflicts between visual and audio results, and can repeatedly identify without relying on the same occluded viewpoint, thus solving the problem of long-term uncertainty in oil leakage detection results when tool carts and material frames in the machine shop obscure the bottom area of ​​the equipment.

[0083] Example 6: Based on all the above examples, the inspection event closed-loop module generates an inspection event confidence value based on visual anomaly evidence, voice anomaly evidence, audio-visual referential association results, and review results, and determines the event handling level accordingly.

[0084] Visual anomalies are multiplied by the reliability of visual acquisition before being included in the inspection event calculation, so that the anomaly image obtained from complete observation has a higher contribution. Speech anomalies are multiplied by the reliability of speech segments before being included in the inspection event calculation, so that speech severely damaged by impact noise will not directly trigger an alarm due to the recognition of a single abnormal word.

[0085] When both visual and audio results point to the same device, the same part of the device, and the same type of anomaly, an audio-visual co-enhancement item is introduced. When the audio report is "oil leak" but the visual anomaly is located in the status light area, no audio-visual co-enhancement item is generated, thus avoiding raising the event level simply because both modalities have anomalies at the same time.

[0086] An abnormal persistence value is generated when the verification result persists. The abnormal persistence value is reduced when the abnormality disappears after verification. The equipment safety level is determined according to the machine tool type and inspection location. The equipment safety level corresponding to the spindle lubrication area, the operating door safety area, and the equipment bottom continuous leakage area is higher than that of the ordinary chip accumulation area.

[0087] Calculate the inspection event confidence value corresponding to the inspection event of the i-th equipment station according to Formula 5:

[0088] In Formula 5, This represents the inspection event confidence value corresponding to the inspection event at the i-th equipment station. This indicates the equipment safety level coefficient corresponding to the current equipment location. Indicates the weight of evidence of visual abnormalities. Indicates the reliability of visual data acquisition. Indicates visual outliers, Indicates the weight of evidence of abnormal speech. Indicates the reliability of a speech segment. Indicates the degree of speech abnormality. Indicates the weights for enhanced audio-visual collaboration. This indicates that the audio-visual references are associated values. Indicates the weight of the review results. This indicates the degree of persistence of the abnormality before and after the review.

[0089] In this embodiment, events with a confidence value below 0.35 are saved as normal inspection records, events with a confidence value between 0.35 and 0.65 are entered into the re-verification event queue, and events with a confidence value of 0.65 are generated as equipment abnormal events. When an operating door is not closed or there is continuous oil leakage and the equipment safety level reaches a high-risk level, on-site images and voice text are sent to the equipment maintenance terminal through the workshop wireless network.

[0090] When the visual anomaly value is high but there is no voice report from the staff, the inspection vehicle can still form an equipment anomaly event based on visual evidence. When the staff's voice report is clear but the current target area is severely obstructed, the voice evidence can trigger the equipment anomaly verification module to generate a new observation position, but the fault cannot be directly identified based solely on low-reliability voice.

[0091] When vision and voice coexist and the audio-visual referencing correlation value is high, the audio-visual collaboration enhancement item increases the confidence value of the inspection event, enabling the local anomalies pointed out by the staff to be reviewed first. When vision and voice point to different devices, the audio-visual collaboration enhancement item automatically decreases, without generating erroneous cross-device anomaly fusion.

[0092] Regarding parameter adjustments: Increase the weight of visual anomaly evidence during unattended nighttime inspections; increase the weight of voice anomaly evidence when day shift operators provide frequent on-site feedback; reduce the safety level of equipment corresponding to abnormal status lights during equipment maintenance; and reload the safety level parameters for the normal production phase after the equipment resumes production.

[0093] Through the above processing, the inspection event closed-loop module does not simply weight the visual recognition results and the voice recognition results. Instead, it uses the consistency of equipment objects, the consistency of equipment parts, and the continuity of verification to determine whether the two modalities form a synergistic enhancement, thus solving the problem of visual anomalies and voice anomalies being incorrectly merged when they belong to different equipment.

[0094] Example 7: This example is based on all the above examples and illustrates the actual operation of the present invention using the daytime inspection process of a precision machining workshop in a machinery manufacturing company.

[0095] The inspection vehicle starts from the workshop entrance and inspects 32 equipment stations in sequence along the established route. The normal route cruising speed is set to 0.6 m / s, which is reduced to 0.25 m / s 2 meters before entering the equipment station. The vehicle stops moving during the visual acquisition of the equipment station. The pan-tilt camera is installed at a height of 1.18 m with a horizontal field of view of 90°. A four-channel microphone array is installed below the camera and faces the front of the inspection vehicle.

[0096] After the inspection vehicle arrived at machining center number 12, it identified the workstation sign for equipment number 12. The leakage area at the bottom of the equipment was obscured by a tool cart with a width of 0.46m. In the initial image, the visible proportion of the target visual inspection area was only 0.56. Therefore, the leakage area at the bottom of the equipment was marked as a visual sample to be reviewed.

[0097] At this moment, a worker stands between machining center number 12 and machining center number 13, points to the lower right of machining center number 12 and says, "There was oil in the lower right corner of this machine just now." The voice text does not contain the equipment number, so the audio-visual reference module simultaneously reads the direction of the worker's voice source, the visual azimuth of machining centers number 12 and 13, the ray pointing from the worker's arm, and the leakage area at the bottom of the equipment corresponding to "lower right corner".

[0098] The deviation between the direction of the worker's voice source and the visual azimuth of machining center No. 12 is 7.6°, and the deviation between the visual azimuth of machining center No. 13 is 24.3°. The distance between the worker's pointing ray and the center of the leakage area at the bottom of machining center No. 12 is 0.12m, and the distance between the worker's pointing ray and the center of the corresponding area at machining center No. 13 is 0.71m. Machining center No. 12 has the highest audiovisual referential correlation value. Therefore, the current abnormal speech semantics is bound to the leakage area at the bottom of machining center No. 12.

[0099] The initial visual acquisition was obstructed by the tool cart, and only small, continuous dark patches were detected. The voice abnormality was high, while the visual abnormality was in the middle range. Therefore, the equipment abnormality verification module was activated to remove the obstruction and complete the verification.

[0100] The tool cart is about 0.80m away from the camera, and the leakage area at the bottom of the No. 12 machining center is about 1.60m away from the camera. After adding a visual safety margin of 0.05m at the edge of the obstruction, the required lateral verification movement distance of about 0.56m is obtained according to Formula 4. After deducting the chassis safety distance, the right side of the inspection vehicle has 0.63m of usable lateral space. Therefore, the chassis is moved 0.56m to the right and re-aligned with the No. 12 machining center.

[0101] The location was re-identified as the workstation sign for equipment number 12. The visibility of the leakage area at the bottom of the equipment increased to 0.91. After perspective correction, continuous oil patches were detected at the bottom of the equipment, with an actual area of ​​approximately 34 cm². 2 Furthermore, the edges of the patches expanded outwards along the liquid outlet at the bottom of the device during three consecutive observation times, thus significantly increasing the visual anomaly value.

[0102] The abnormal location in the reviewed image is consistent with the staff's voice recording of "oil in the lower right corner". The audio-visual collaboration enhancement item and the abnormality persistence value increase synchronously. The inspection event confidence value reaches the equipment abnormality event condition. The inspection vehicle writes the No. 12 machining center, the leakage area at the bottom of the equipment, the oil leakage, the on-site image, the staff's voice text, the images before and after review, and the inspection time into the equipment abnormality event.

[0103] The inspection vehicle then arrived at CNC lathe No. 16. There were bright spots on the ground under the equipment caused by strong light reflection. The initial visual identification produced a low-level leakage candidate result, but there was no staff member to report it verbally. After the inspection vehicle changed the angle of the pan-tilt head, the position of the bright spot moved significantly with the observation angle and the continuous boundary disappeared. Therefore, the visual anomaly value decreased after verification. The current result is only saved as a normal inspection record.

[0104] Near machining center number 21, a worker stood in front of adjacent equipment number 22 and said, "This door doesn't seem to be closed properly." The direction of the microphone sound source and the worker's pointing ray both pointed to equipment number 22. Even though equipment number 21 was closer to the inspection vehicle, the audio-visual association module still bound the voice to equipment number 22 to avoid incorrect association based on the closest distance.

[0105] When compressed air is injected instantaneously during the inspection, the proportion of impact noise frames in the affected voice segment increases, and the reliability of the voice segment drops to the range to be confirmed. Instead of generating a fault using the damaged voice recognition result, the inspection vehicle asks the staff to explain again through the loudspeaker. After the second voice segment meets the valid voice conditions, it re-enters the audiovisual reference association process.

[0106] Based on the above-mentioned actual working conditions, this invention uses visual acquisition reliability to determine whether the current image has the conditions for anomaly recognition, uses voice segment reliability to suppress the misinterpretation of feedback from workers by noise in the machine shop, uses audiovisual referential correlation value to determine the actual equipment and inspection location pointed to by the voice, and actively obtains new visual evidence by removing occlusion and verifying the posture. Finally, it forms a closed-loop inspection event for specific equipment, specific location and specific anomaly, thereby improving the anomaly recognition accuracy and on-site usability of intelligent inspection vehicles in complex occlusion and high noise environments in machine shops.

Claims

1. A mechanical workshop intelligent inspection vehicle based on visual and voice collaborative perception, comprising a mobile chassis, a workstation positioning module, an equipment visual anchoring module, a workshop voice perception module, a sound and visual reference association module, an equipment anomaly verification module, and an inspection event closed-loop module; The workstation positioning module acquires the equipment workstation identification plate, determines the equipment identity and workstation position of the currently inspected equipment based on the equipment workstation identification plate, generates the equipment workstation positioning result, and sends the equipment workstation positioning result to the equipment visual anchoring module. The equipment visual anchoring module receives the equipment workstation positioning result, establishes an equipment workstation coordinate system based on the equipment workstation identification sign, maps the standard inspection template corresponding to the equipment identity to the current acquisition screen, generates a visual inspection area, generates visual acquisition reliability based on the visibility status and image acquisition status of the visual inspection area, and performs anomaly identification on the visual inspection area to generate visual anomaly results. The workshop voice perception module collects the background sound of the equipment at the current equipment station and the voice of the staff. It establishes a noise baseline for the equipment based on the background sound, processes the voice of the staff based on the noise baseline, obtains the voice text of the staff and the direction of the sound source of the staff, and generates the reliability of the voice segment and the voice perception result based on the noise impact state of the voice of the staff. The audio-visual association module receives the visual inspection area, the visual acquisition reliability, the staff's sound source direction, the speech segment reliability, and the speech perception result. It generates an audio-visual association value based on the matching relationship between the staff's sound source direction, the staff's pointing direction, the speech perception result, and the candidate device. It then generates a target device association result based on the audio-visual association value. The device anomaly verification module receives the target device association result, the visual anomaly result, and the visual acquisition reliability. When there is occlusion in the target visual inspection area and the visual anomaly result is inconsistent with the voice perception result, it generates a verification pose based on the position of the occluder, the position of the target visual inspection area, and the available space of the inspection channel. It controls the mobile chassis to move to the verification pose and reacquires the target visual inspection area to generate the device anomaly verification result. The inspection event closed-loop module receives the visual anomaly result, the visual acquisition reliability, the voice perception result, the voice segment reliability, the target device association result, and the device anomaly review result. Based on the visual anomaly evidence, the voice anomaly evidence, the consistency status of the audio-visual objects, and the anomaly persistence status before and after review, it generates an inspection event confidence value and generates an inspection event result based on the inspection event confidence value.

2. The intelligent inspection vehicle for a mechanical workshop based on visual and voice collaborative perception as described in claim 1, wherein the equipment visual anchoring module establishes the equipment workstation coordinate system according to the actual size and image position of the equipment workstation identification sign, and generates a three-color status light area, an operating door closing area, a bottom leakage area of ​​the equipment, a chip discharge port area, and an instrument display area according to the standard inspection template; the equipment visual anchoring module generates the visual acquisition reliability based on the equipment identification confidence level, the proportion of the unobstructed area of ​​the visual inspection area, the degree of image motion blur, and the position mapping error of the standard inspection template, and determines the corresponding visual inspection area as a valid visual sample and a visual sample to be verified based on the visual acquisition reliability.

3. The intelligent inspection vehicle for a mechanical workshop based on visual and voice collaborative perception as described in claim 2, wherein the workshop voice perception module includes a microphone array, wherein the microphone array collects the background sound of equipment operation and establishes the equipment noise baseline after the mobile chassis enters the current equipment workstation, divides the worker's voice into multiple analysis frames, generates the reliability of the voice segment based on the number of impact noise frames in the analysis frames, the relationship between the effective frequency band energy of the voice and the background noise energy of the equipment, and the duration of no effective voice, and extracts the equipment number, equipment location, abnormal action information and spatial description information from the worker's voice text to generate the voice perception result.

4. The intelligent inspection vehicle for a mechanical workshop based on visual and voice collaborative perception as described in claim 3, wherein the audio-visual reference association module converts the direction of the worker's voice source to the camera's observation coordinate system, generates the worker's pointing direction based on the positions of the worker's shoulder, elbow, and wrist, and generates the audio-visual reference association value corresponding to each candidate device based on the difference between the worker's voice source direction and the visual orientation of each candidate device, the positional relationship between the worker's pointing direction and the visual inspection area of ​​each candidate device, and the semantic matching relationship between the voice perception result and the device identity and device part of each candidate device; the audio-visual reference association module generates the target device association result for the candidate device corresponding to the highest audio-visual reference association value, and generates a reconfirmation request when the difference between the highest audio-visual reference association value and the second highest audio-visual reference association value does not reach the target differentiation condition.

5. The intelligent inspection vehicle for a mechanical workshop based on visual and voice collaborative perception as described in claim 4, wherein the equipment anomaly verification module acquires the lateral width of the obstruction obscuring the target visual inspection area, the distance from the obstruction to the camera, the distance from the target visual inspection area to the camera, and the available lateral movement distance on both sides of the mobile chassis; generates the lateral verification displacement required to remove the obstruction based on the lateral width of the obstruction, the distance from the obstruction to the camera, and the distance from the target visual inspection area to the camera; limits the lateral verification displacement according to the available lateral movement distance; and controls the mobile chassis to perform lateral movement to re-identify the equipment workstation sign and the target visual inspection area.

6. The intelligent inspection vehicle for mechanical workshops based on visual and voice collaborative perception as described in claim 5, wherein the equipment visual anchoring module generates the visual anomaly result based on the actual illumination status of the three-color status light area, the actual gap between the door edge and the door frame baseline of the operating door closing area, the actual area and area change of the liquid patch in the equipment bottom leakage area, the actual height of the chip accumulation in the chip discharge port area, and the instrument status of the instrument display area; for the equipment bottom leakage area, the equipment anomaly verification module generates the leakage verification result based on the changes in the liquid patch boundary, the actual area change of the liquid patch, and the spatial relationship between the liquid patch and the liquid outlet position at the bottom of the equipment during continuous observation.

7. The intelligent inspection vehicle for a mechanical workshop based on visual and voice collaborative perception as described in claim 6, wherein the inspection event closed-loop module generates visual anomaly evidence based on the reliability of visual acquisition and visual anomaly results, generates voice anomaly evidence based on the reliability of voice segments and voice perception results, generates a consistent state of audio-visual objects based on the audio-visual referential association value, generates an anomaly persistence state before and after verification based on the equipment anomaly verification results, and generates the inspection event confidence value in combination with the equipment safety level corresponding to the current equipment location; when the visual anomaly evidence and the voice anomaly evidence correspond to the same equipment identity and the same equipment location, the contribution of audio-visual collaborative evidence is increased; when the visual anomaly evidence and the voice anomaly evidence correspond to different equipment identities and different equipment locations, the generation of audio-visual collaborative evidence contribution is stopped.

8. The intelligent inspection vehicle for a mechanical workshop based on visual and voice collaborative perception as described in claim 7, wherein the inspection event results include normal inspection records, re-verification events, and equipment abnormality events; the inspection event closed-loop module determines the corresponding inspection event result based on the inspection event confidence value, and associates and stores the equipment abnormality event with the equipment number, abnormal part, abnormality type, on-site image, staff voice text, inspection time, and the equipment abnormality re-verification result; after generating the inspection event result corresponding to the current equipment station, the mobile chassis moves to the next equipment station and continues to perform equipment inspection.

Images