Abnormality identification method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses an abnormality identification method and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring an acoustic imaging cloud picture collected when a target device plays target audio; performing feature extraction on the acoustic imaging cloud picture to obtain acoustic feature information; and identifying abnormal type information of the target device according to the high acoustic feature information.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and specifically to an anomaly recognition method, apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] In the research and development and manufacturing process of audio equipment, it is usually necessary to test the sound performance of the audio equipment to confirm whether there are any abnormal sound phenomena when the audio equipment plays audio. Acoustic abnormalities such as vibration and air leakage are relatively common types of abnormalities in audio equipment. These abnormalities may occur at different stages, such as prototype development, structural verification, process debugging, and mass production testing.

[0003] In actual production and R&D testing, the identification of abnormal phenomena such as vibration and air leakage usually relies on testers or experienced engineers to judge by listening. Engineers confirm the existence and approximate type of abnormality based on the differences in sound perception when the audio device plays specific audio. This process depends to some extent on human experience and requires a significant investment of time and effort. Summary of the Invention

[0004] This application provides an anomaly identification method, apparatus, electronic device, and computer-readable storage medium.

[0005] In a first aspect, embodiments of this application provide an anomaly identification method, the method comprising: Acoustic imaging cloud images were captured when the target device played the target audio. Acoustic feature information is obtained by extracting features from the acoustic imaging cloud map; The abnormality type information of the target device is identified based on the acoustic feature information.

[0006] Secondly, embodiments of this application also provide an anomaly detection device, the device comprising: The acquisition module is used to acquire acoustic imaging cloud images collected when the target device plays the target audio; The extraction module is used to extract acoustic feature information from the acoustic imaging cloud map; The identification module is used to identify the abnormality type information of the target device based on the acoustic feature information.

[0007] Thirdly, embodiments of this application also provide an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps in the above-described anomaly identification method.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in the above-described anomaly identification method.

[0009] Fifthly, embodiments of this application also provide a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described in embodiments of this application. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram of a scenario in which a terminal device, as provided in an embodiment of this application, executes the anomaly identification method; Figure 2 This is a flowchart illustrating the anomaly identification method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the anomaly detection device provided in the embodiments of this application; Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application.

[0012] Explanation of icon numbers: 101-Terminal device; 301-Acquisition module; 302-Extraction module; 303-Identification module; 401-Processor; 402-Memory; 403-Power supply; 404-Input unit. Detailed Implementation

[0013] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] It should be understood that the following embodiments are only used to explain this application and are not intended to limit the scope of protection of this application. Where there is no conflict, the various technical features in the embodiments of this application can be combined with each other.

[0015] This application provides an anomaly identification method, apparatus, electronic device, and computer-readable storage medium. Specifically, this application provides an anomaly identification apparatus suitable for electronic devices, including terminal devices such as mobile phones, tablets, laptops, or televisions. The anomaly identification method provided in this application can be applied to anomaly detection scenarios for audio devices, such as speakers, enclosures, headphones, electroacoustic transducers, or other target devices capable of playing audio and generating / outputting sound.

[0016] For example, please see Figure 1 , Figure 1 This is a schematic diagram illustrating a scenario where a terminal device, according to an embodiment of this application, executes the anomaly detection method. Specifically, the execution process of the anomaly detection method by the terminal device is as follows: Terminal device 101 acquires an acoustic imaging cloud image collected when target device 102 plays target audio, extracts features from the acoustic imaging cloud image to obtain acoustic feature information, and identifies the abnormal type information of the target device based on the acoustic feature information.

[0017] For example, the target device is controlled to play a specific target audio, and then the acoustic signal is collected by an acoustic camera and an acoustic imaging cloud map is output. Then, feature extraction and anomaly type identification are performed on the acoustic imaging cloud map.

[0018] It is understandable that by acquiring acoustic imaging cloud images when the target device plays the target audio, and performing feature extraction and anomaly type identification based on the acoustic imaging cloud images, compared with simply relying on manual listening, the embodiments of this application can improve the objectivity and consistency of anomaly type identification results, reduce the impact of differences in human experience on detection results, and improve the efficiency of anomaly detection of the target device.

[0019] The following sections provide detailed descriptions of each example. It should be noted that the order in which the embodiments are described is not intended to limit the priority of the embodiments.

[0020] Please see Figure 2 , Figure 2 This is a flowchart illustrating the anomaly identification method provided in this application. Although the flowchart shows a logical order, in some cases, the steps shown or described can be performed in a different order than that shown in the flowchart. Specifically, the specific flow of the anomaly identification method is as follows: S201. Acquire the acoustic imaging cloud map collected when the target device plays the target audio.

[0021] The target device refers to the device to be detected for anomalies, including speaker units, speaker modules, speaker cabinets, headphones, or other audio playback devices. In this embodiment, the target device is not a detection device for acquiring acoustic imaging cloud images, but rather plays target audio as the object to be detected.

[0022] The target audio is the test audio played by the target device during the anomaly identification process. For example, the target audio can be a sweep signal, a single-frequency signal, pink noise, white noise, a preset music clip, a pulse signal, a test sound source containing a specific frequency band, or an audio signal used to excite abnormal sound phenomena such as vibration or air leakage. In this embodiment, the target audio can be pre-configured according to the target device type, test purpose, type of anomaly to be identified, or production line test specifications.

[0023] The "play target audio" setting instructs the target device to output the sound corresponding to the target audio. For example, the target audio can be input to the target device via an audio playback control device, power amplifier, testing software, or production line control system, causing the target device to output sound according to a preset volume, preset frequency range, preset playback duration, or preset playback order.

[0024] The acoustic imaging cloud map is used to indicate image information obtained through acoustic imaging that characterizes the spatial distribution of the sound field. For example, the acoustic imaging cloud map can be a color heatmap generated by an acoustic camera based on multi-channel acoustic signals acquired by a microphone array, where different colors in the color heatmap represent different sound intensities, sound pressure levels, or sound energy levels. In the embodiments of this application, the acoustic imaging cloud map can also be a normalized acoustic image such as a grayscale cloud map, a sound source localization map, or a sound energy distribution map. It is understood that the acoustic imaging cloud map can include information such as the spatial location of abnormal sound sources in the cloud map, sound intensity distribution, sound energy concentration areas, hotspot areas, and sound field diffusion patterns.

[0025] In this embodiment of the application, external acoustic detection equipment can be used to acquire acoustic data when the target device plays target audio and generate an acoustic imaging cloud map. For example, an acoustic camera can be used to collect the sound output when the target device plays target audio, multi-channel sound signals can be obtained through a microphone array, and the sound source location and intensity can be calculated based on a beamforming algorithm. Then, the sound intensity distribution can be visualized into an acoustic imaging cloud map.

[0026] That is, in this step, the target device can be controlled to play the target audio first, and then the acoustic camera can be used to collect the sound field distribution information corresponding to the target device during the playback of the target audio, thereby generating and acquiring the acoustic imaging cloud map corresponding to the target audio. Alternatively, when performing anomaly detection on the target device, the acoustic imaging cloud map generated by the target device playing the target audio in advance can be acquired.

[0027] Understandably, since acoustic imaging cloud maps can present the spatial distribution of the sound field when the target device plays the target audio, this step can preserve the location distribution information of abnormal sound sources, providing a data basis for subsequent anomaly type identification and anomaly location determination.

[0028] S202. Perform feature extraction on the acoustic imaging cloud map to obtain acoustic feature information.

[0029] Feature extraction refers to the process of extracting features that characterize the acoustic state of a target device from an acoustic imaging cloud image. For example, feature extraction may include image grayscale feature extraction, color distribution feature extraction, texture feature extraction, acoustic hotspot region extraction, sound intensity gradient feature extraction, sound field energy concentration degree extraction, abnormal region boundary feature extraction, and spatial geometric relationship feature extraction.

[0030] Among them, acoustic feature information is used to indicate information that can characterize the acoustic spatial distribution state when the target device plays the target audio, obtained by feature extraction from acoustic imaging cloud maps. For example, the acoustic feature information may include spatial distribution information of sound intensity, spatial distribution information of sound pressure level, spatial distribution information of sound energy, location features of abnormal sound sources, area features of acoustic hotspots, shape features of acoustic hotspots, sound field texture features, sound field diffusion direction features, and sound intensity difference features between different regions.

[0031] Among them, the acoustic spatial distribution state is used to indicate the spatial distribution of sound energy, sound intensity, sound pressure level, or abnormal sound sources within the detection area. For example, when the target equipment has an air leakage anomaly, the acoustic imaging cloud map may show local high-intensity hot spots at locations such as structural joints, screw holes, and adhesive seams; when the target equipment has an abnormal vibration sound, the acoustic imaging cloud map may show abnormal patterns or local energy concentration areas related to the structural vibration area.

[0032] Understandably, because different pixels in an acoustic imaging cloud image have spatial relationships, acoustic feature information can reflect not only the intensity of abnormal sounds, but also their spatial location, extent, and distribution within the target device or detection area. Therefore, it facilitates subsequent identification of the anomaly type of the target device based on acoustic feature information.

[0033] S203. Identify the abnormality type information of the target device based on the acoustic feature information.

[0034] The identification process is used to indicate the steps involved in determining the output based on the input information. For example, acoustic feature information can be processed using preset rules, classification models, image recognition models, neural network models, or probabilistic reasoning models to determine whether the target device is abnormal and what kind of abnormality it is.

[0035] The anomaly type information is used to indicate the anomaly category or abnormal sound type of the target device. For example, in the embodiments of this application, the anomaly type information may include one or more of the following types: vibration, air leakage, wind noise, abnormal sound pressure, abnormal noise, loose structural components, abnormal assembly gaps, abnormal cavity sealing, etc.

[0036] In the embodiments of this application, the anomaly type information can be represented by a category name, category number, probability vector, confidence result, or multi-label recognition result. For example, the anomaly type information can be represented as "air leakage, confidence 0.86", or as "vibration, probability 0.72, air leakage, probability 0.18, wind noise, probability 0.10", etc.

[0037] In the embodiments of this application, the abnormality type information of the target device can be identified based on the features contained in the acoustic feature information, such as the spatial distribution of sound intensity, the morphology of abnormal areas, the location of acoustic hotspots, and changes in cloud map texture. For example, when the abnormal sound intensity in the acoustic imaging cloud map is concentrated near the screw hole and has a local jet-like or point-like high-energy distribution, it can be identified as an air leakage anomaly; when the acoustic imaging cloud map shows irregular diffusion or vibration-related textures in a certain structural component area, it can be identified as a vibration-related anomaly, etc.

[0038] In summary, this application embodiment acquires acoustic imaging cloud images when the target device plays target audio, and performs feature extraction and anomaly type identification based on the acoustic imaging cloud images. Compared with the method of simply relying on manual listening, this application embodiment can improve the objectivity and consistency of anomaly type identification results, reduce the impact of differences in human experience on detection results, and improve the efficiency of target device anomaly detection.

[0039] In this application embodiment, to further enable the anomaly identification results (anomaly type information) to be quickly used for subsequent maintenance, debugging, or process optimization, in addition to determining the type of anomaly, the location of the anomaly on the target equipment can be further determined, and the corresponding target component and optimization indication information can be determined based on the location of occurrence, so as to quickly perform maintenance and adjustment, etc. That is, optionally, in some embodiments of this application, the method further includes: Anomaly location information is determined on the acoustic imaging cloud map based on the acoustic feature information, wherein the anomaly location information is the location of the anomaly corresponding to the anomaly type information on the acoustic imaging cloud map; Based on the abnormal location information, target component information for the target device is determined, and the target component information is used to indicate the device components on the target device; The optimization instruction information corresponding to the anomaly type information is determined based on the target component information.

[0040] Among them, the abnormal location information is used to indicate the location of the abnormality corresponding to the abnormality type information in the acoustic imaging cloud map. The abnormal location information can be pixel coordinates, coordinate regions, bounding boxes, segmentation masks, cloud map partition numbers, or acoustic hotspot center points in the acoustic imaging cloud map. For example, the abnormal location information can be "the second region of the cloud map", "the bounding box corresponding to coordinates (x1, y1, x2, y2)", "the high acoustic intensity hotspot in the region corresponding to the screw hole", etc.

[0041] The target component information is used to indicate equipment components on the target device. For example, target component information may include the name, number, or location identifier of equipment components such as screw holes, panels, housing seams, speaker drivers, sound tubes, sealing rings, adhesive seams, back covers, grilles, cavities, and structural supports. It can be understood that the target component information is used to map abnormal locations in the acoustic imaging cloud image to specific structural components of the target device.

[0042] The optimization instruction information is used to indicate the direction of adjustment, repair, assembly, or process optimization determined based on the anomaly type information and / or target component information. For example, in this embodiment, the optimization instruction information may include tightening screws, checking screw hole seals, adding EVA stickers, adjusting sealant, checking structural joints, reassembling panels, checking cavity seals, adjusting target component mating clearances, and replacing abnormal components. In this embodiment, the optimization instruction information may be output in the form of text suggestions, measure numbers, processing flows, process parameter adjustment information, or maintenance instructions.

[0043] In this embodiment, based on acoustic feature information, abnormal location information is first located on the acoustic imaging cloud map. Then, this abnormal location information is mapped to a specific component of the target device. Finally, optimized instruction information is generated for that component / target device. For example, when the abnormality type information is air leakage and the abnormal location information corresponds to the screw hole area of ​​the target device, the target component information can be determined to be a screw hole or screw connection structure. Therefore, the optimized instruction information can be determined to be to check whether the screw is tightened or to check the sealing status of the screw hole. Similarly, when the abnormality type information is air leakage and the abnormal location information corresponds to the structural joint area, the target component information can be determined to be a structural joint. Therefore, the optimized instruction information can be determined to be to add a sealant or apply an EVA sticker, etc.

[0044] In summary, in this embodiment, not only is the abnormality type information of the target device identified based on acoustic feature information, but the abnormality location information is also located on the acoustic imaging cloud map based on the acoustic feature information, and the abnormality location information is mapped to a specific component of the target device and associated with the specific component, thereby facilitating the provision of more accurate optimization instructions for the specific abnormal component.

[0045] Optionally, in this embodiment, when determining the target component information based on the abnormal location information, it can be determined by pre-establishing a correspondence between the component position of the target device and the position in the acoustic imaging cloud map, or it can be automatically determined by using the coordinate registration relationship between the device image and the acoustic imaging cloud map. That is, optionally, in some embodiments of this application, the step "locating the target component information on the target device based on the abnormal location information" includes: The abnormal location information is determined to be located in the cloud map partition on the acoustic imaging cloud map, and the device component information corresponding to the cloud map partition is determined as the target component information, wherein the cloud map partition and the device component information of the target device correspond one-to-one; And / or, Acquire a device image when the target device plays the target audio, register the coordinates of the device image with the acoustic imaging cloud map, and determine the target component information based on the abnormal location information and the device image.

[0046] Among them, cloud map partitioning is the sub-region formed by dividing the acoustic imaging cloud map into regions. For example, the acoustic imaging cloud map can be divided into upper left partition, upper right partition, lower left partition, and lower right partition, or it can be divided according to the distribution of structural components of the target device into screw hole partition, panel seam partition, speaker unit partition, sound tube partition, etc.

[0047] Among them, equipment component information is used to indicate the identification information of each component in the target equipment. For example, equipment component information may include component name, component number, component area coordinates, component outline, component corresponding cloud map partition, etc.

[0048] The one-to-one correspondence refers to the pre-established correspondence between cloud map partitions and equipment component information. For example, for a fixed product in a specific production run, an acoustic imaging cloud map of a product without abnormalities can be collected first. This acoustic imaging cloud map can then be manually divided into multiple partitions, with each partition corresponding to a specific equipment component or area of ​​a target equipment. Subsequently, during anomaly detection, if the anomaly location information falls into a certain cloud map partition, the equipment component information corresponding to that cloud map partition can be identified as the target component information.

[0049] Among them, equipment images are used to indicate images that can present the appearance or structural location of a target device. For example, equipment images can be target device images captured by visible light cameras, production station camera images, front views of target devices, partial structural diagrams of target devices, or optical images corresponding to acoustic imaging cloud maps, etc.

[0050] Coordinate registration refers to the alignment of positions between the equipment image and the acoustic imaging cloud map. For example, abnormal positions in the acoustic imaging cloud map can be mapped to corresponding positions in the equipment image through methods such as camera calibration, spatial coordinate transformation, image alignment, feature point matching, and fixed shooting posture calibration at the same workstation.

[0051] In one implementation, a device-free imaging approach can be used. This involves pre-dividing the acoustic imaging cloud map into multiple cloud map partitions and establishing a correspondence between these partitions and the various components of the target device. This method is suitable for production line scenarios where product models are fixed, test postures are fixed, and the relative positions of the acoustic camera and the target device are fixed. It can achieve the correspondence between abnormal locations and target components at a relatively low cost.

[0052] In another implementation, a device image can be acquired when the target device is playing the target audio, and this device image can be coordinately registered with an acoustic imaging cloud map. Since the device image can show the actual location of the target device's components, and the acoustic imaging cloud map can show the location of abnormal sound intensities, the target component information can be determined based on the coordinates of the abnormal location information in the acoustic imaging cloud map and the spatial location of the component in the device image. This method is applicable to devices with different products, different placement postures, or different structural layouts, improving the automation level and applicability of the target component determination process.

[0053] Optionally, after determining the target component information, the combination of the anomaly type information and the target component information can further reflect the cause of the anomaly. For example, even with the same anomaly of air leakage, the corresponding causes and handling measures may differ depending on whether it occurs at a screw hole or at a structural joint. Therefore, the optimized indication information can be determined by combining the target component information and the anomaly type information, or the anomaly type information can be corrected using the target component information before determining the optimized indication information. That is, optionally, in some embodiments of this application, the step "determining the optimized indication information corresponding to the anomaly type information based on the target component information" includes: The optimization instruction information is determined based on the target component information and the anomaly type information; And / or, The abnormality type information is corrected based on the target component information to obtain corrected abnormality type information, and the optimization indication information is determined based on the target component information and the corrected abnormality type information.

[0054] Correction refers to the process of re-evaluating or adjusting the probability of anomaly type information based on target component information. For example, if the initial identification shows an air leakage probability of 0.55 and a vibration probability of 0.40, but the anomaly location corresponds to a component area prone to structural vibration, the probability of the vibration type can be increased based on the target component information to obtain corrected anomaly type information.

[0055] The corrected anomaly type information indicates the anomaly type result after adjusting the initial anomaly type information obtained using target component information. For example, the corrected anomaly type information can be the corrected anomaly category, the corrected anomaly type probability vector, or the reordered anomaly type candidate results.

[0056] That is, in one possible implementation, optimization instructions can be directly determined based on the target component information and the anomaly type information. For example, if the target component information is a screw hole and the anomaly type information is air leakage, the optimization instructions could be to check if the screws are tightened and if the screw hole seal is intact; if the target component information is a panel seam and the anomaly type information is air leakage, the optimization instructions could be to check the structural seam and add a seal; if the target component information is a housing area and the anomaly type information is vibration, the optimization instructions could be to check the housing assembly gap or the looseness of structural components.

[0057] In another possible implementation, the anomaly type information can be corrected based on the target component information. For example, different equipment components correspond to different common anomaly types; screw holes, seams, and seals are more prone to air leakage anomalies, while panels, housings, and brackets are more prone to vibration anomalies. By introducing target component information, the anomaly type identification results can be made more consistent with the actual structural scenario. Subsequently, optimized indication information can be determined based on the target component information and the corrected anomaly type information, thereby improving the matching degree between the optimized indication information and the actual cause of the anomaly.

[0058] Optionally, to improve the automation level of anomaly identification, anomaly localization, target component determination, and optimized indication information generation, an image recognition model can be used to process the acoustic imaging cloud map and / or device image. That is, optionally, in some embodiments of this application, the step "extracting features from the acoustic imaging cloud map to obtain acoustic feature information" includes: The acoustic feature information is obtained by extracting the feature information of the acoustic imaging cloud map through an image recognition model; And, it also includes: The image recognition model identifies the anomaly type information corresponding to the acoustic feature information and determines the anomaly location information corresponding to the anomaly type information. The target component information is determined by the image recognition model based on the abnormal location information and the device image; The image recognition model corrects the anomaly type information based on the target component information to obtain the corrected anomaly type information; The optimization indication information is determined by the image recognition model based on the target component information, the anomaly type information, and / or the corrected anomaly type information.

[0059] Image recognition models are used to refer to models capable of performing feature extraction, classification, localization, detection, segmentation, or output of results for images. For example, image recognition models can be convolutional neural network models, two-dimensional convolutional neural network models, three-dimensional convolutional neural network models, object detection models, semantic segmentation models, instance segmentation models, multi-task learning models, or neural network models that fuse acoustic imaging cloud maps and device images, etc.

[0060] In this context, extracting feature information through an image recognition model refers to inputting an acoustic imaging cloud map into the image recognition model, and having the model's convolutional layers, feature pyramids, encoders, or other feature extraction structures output acoustic feature information. For example, this image recognition model can extract sound intensity texture features, acoustic hotspot morphology features, anomalous region boundary features, and spatial distribution features of sound field energy from the acoustic imaging cloud map.

[0061] In this embodiment, the image recognition model can perform multiple tasks: first, extract acoustic feature information from the acoustic imaging cloud image; second, identify anomaly type information based on the acoustic feature information; third, determine the anomaly location information corresponding to the anomaly type information; fourth, determine the target component information based on the anomaly location information and the device image; fifth, correct the anomaly type information based on the target component information; and sixth, determine optimization indication information based on the target component information, the anomaly type information, and / or the corrected anomaly type information.

[0062] For example, an acoustic imaging cloud image is input into an image recognition model, which outputs anomaly type information and probability, as well as anomaly region bounding boxes (anomaly location information). Then, combined with the identified target device component locations in the device image (spatial location information), the device component corresponding to the anomaly region is determined (target component information). Finally, based on the combination of "device component + anomaly type," optimization indication information is output. Thus, the image recognition model can integrate anomaly type identification, anomaly location determination, and optimization indication information determination into an automated processing flow, achieving anomaly type identification, localization, and optimization indication.

[0063] Optionally, in this embodiment of the application, in order to enable the image recognition model to process acoustic imaging cloud maps and device images respectively, and to realize the associated output of anomaly type, anomaly location, target component and optimization indication information, the image recognition model can be set as a model structure including multiple functional modules. That is, optionally, in some embodiments of this application, the image recognition model includes a first feature extraction module, a first recognition module, a second feature extraction module, a second recognition module, a positioning module, an optimization module and an output module. The input terminal of the first feature extraction module is configured to receive the acoustic imaging cloud map, the output terminal of the first feature extraction module is connected to the input terminal of the first recognition module, and the output terminal of the first recognition module is connected to the input terminal of the positioning module. The input terminal of the second feature extraction module is configured to receive the device image, the output terminal of the second feature extraction module is connected to the input terminal of the second recognition module, and the output terminal of the second recognition module is connected to the input terminal of the positioning module. The output of the positioning module is connected to the input of the optimization module, and the output of the optimization module is connected to the input of the output module. The output of the output module is configured to output the target component information, the anomaly type information, the corrected anomaly type information, or the optimization indication information.

[0064] Specifically, the first feature extraction module is used to extract features from the acoustic imaging cloud map to obtain acoustic feature information; The first identification module is used to identify the acoustic feature information to obtain the anomaly type information and the anomaly location information; The second feature extraction module is used to extract features from the device image to obtain spatial structure feature information of the device; The second identification module is used to identify the spatial structure feature information of the device to obtain the spatial position information of each device component of the target device; The positioning module is used to locate the target device component from each of the device components of the target device according to the abnormal location information and the spatial location information, and to obtain the target component information; The optimization module is used to correct the anomaly type information according to the target component information to obtain the corrected anomaly type information, and to determine the optimization instruction information according to the target component information, the anomaly type information and / or the corrected anomaly type information; The output module is used to output the target component information, the anomaly type information, the corrected anomaly type information, or the optimization indication information.

[0065] The first feature extraction module is used to indicate the model part that performs feature extraction on the acoustic imaging cloud map. For example, the first feature extraction module may include a two-dimensional convolutional layer, a three-dimensional convolutional layer, a residual network, a feature pyramid network, or a visual Transformer encoder, used to extract sound intensity texture features and abnormal sound intensity distribution features from the acoustic color cloud map.

[0066] The first identification module refers to the model part that identifies anomaly type and location information based on acoustic feature information. For example, the first identification module may include a classification head and a detection head. The classification head outputs the probability of anomaly types such as vibration, air leakage, and wind noise, while the detection head outputs the bounding box of the abnormal sound source, the mask of the abnormal region, or the coordinates of the abnormal location.

[0067] The second feature extraction module is the model part that extracts features from the device image. For example, the second feature extraction module can extract spatial structural features of the target device, such as its external outline, component boundaries, structural geometric relationships, screw hole positions, seam positions, and speaker unit positions.

[0068] Among them, equipment spatial structure feature information is used to indicate information in the equipment image that can characterize the structural layout and spatial relationships of the target equipment components. For example, equipment spatial structure feature information may include product outline features, component boundary features, spatial geometric structure features, and the coordinate regions of each equipment component in the image.

[0069] The second recognition module refers to the model part that identifies the spatial position information of each component of the target device based on the spatial structural feature information of the device. For example, the second recognition module can identify components such as screw holes, seams, panels, housings, and speaker units, and output the coordinates or regions of each component in the device image.

[0070] The localization module refers to the model part that determines the target component information based on the abnormal location information and the spatial location information of the equipment components. For example, the localization module can map the abnormal area in the acoustic imaging cloud map to the corresponding area in the equipment image and determine which equipment component's spatial location range the abnormal area falls within.

[0071] The optimization module refers to the model part that corrects the anomaly type information based on the target component information and determines the optimization indication information. For example, the optimization module can combine the target component information and the anomaly type probability to adjust the probability of the anomaly type; or it can determine the optimization indication information based on the target component information, the anomaly type information / corrected anomaly type information, and historical improvement data.

[0072] The output module refers to the model portion that outputs the processing results. For example, the output module can output at least one of the following: target component information, anomaly type information, corrected anomaly type information, and optimization instruction information.

[0073] In summary, through the above modular structure, features can be extracted from both the acoustic imaging cloud image and the device image, and then a correspondence between the location of the abnormal sound source and the location of the device component can be established in the localization module. The first feature extraction module and the first recognition module are used to process acoustic information, the second feature extraction module and the second recognition module are used to process device structural information, the localization module is used to perform cross-image position matching, and the optimization module is used to combine location and anomaly type to correct and optimize indicator generation. Therefore, the image recognition model can simultaneously utilize the abnormal sound intensity information in the acoustic color cloud image and the spatial structural information in the device image, improving the automation level of target component localization and optimization indicator information determination.

[0074] In some embodiments, when the correspondence between abnormal locations and device components of the target device is pre-established through cloud map partitioning or through manual annotation, the image recognition model may not include a second feature extraction module and a second recognition module for processing device images. That is, the image recognition model may include a first feature extraction module, a first recognition module, a component matching module, an optimization module, and an output module.

[0075] The system comprises the following modules: a first feature extraction module receives an acoustic imaging cloud image and extracts features from it to obtain acoustic feature information; a first identification module identifies anomaly type information based on the acoustic feature information and determines the anomaly location information corresponding to the anomaly type information; a component matching module determines the target component information corresponding to the anomaly location based on the anomaly location information and a pre-established correspondence between cloud image partitions and device component information; an optimization module corrects the anomaly type information based on the target component information to obtain corrected anomaly type information, and determines optimization indication information based on the target component information, the anomaly type information, and / or the corrected anomaly type information; and an output module outputs at least one of the target component information, the anomaly type information, the corrected anomaly type information, and the optimization indication information.

[0076] Specifically, with the target device's model, test posture, and acoustic camera acquisition position fixed, a baseline acoustic imaging cloud map can be pre-acquired when the target device plays target audio without any abnormalities. Based on the corresponding positions of the target device's components in the baseline acoustic imaging cloud map, the baseline acoustic imaging cloud map is divided into multiple cloud map partitions. Each cloud map partition corresponds to a component or area of ​​the target device, such as a screw hole area, housing seam area, panel area, speaker unit area, or sealing structure area. Subsequently, during the anomaly identification process, after the image recognition model identifies the anomaly location information based on the acoustic imaging cloud map, the component matching module determines the cloud map partition where the anomaly location information is located and identifies the corresponding component information as the target component information.

[0077] In another example, regions corresponding to each device component can be pre-marked on the acoustic imaging cloud map or a reference acoustic imaging cloud map by humans, forming a mapping relationship between abnormal locations and device component information. After outputting the abnormal location information, the image recognition model determines the target component information corresponding to the abnormal location based on this mapping relationship, or, after determining the abnormal location information, the target component information is determined manually, directly obtaining the target component information input or selected by the user. Thus, without the need to additionally acquire device images or perform coordinate registration between the device images and the acoustic imaging cloud map, the correspondence between abnormal locations and specific target device components can be achieved, thereby simplifying the structure and processing flow of the image recognition model and reducing the computational load.

[0078] Optionally, to enable the image recognition model to accurately identify anomaly types, locate anomaly positions, and output optimization indication information, a sample dataset containing normal and abnormal samples can be pre-constructed, and the initial recognition model can be trained using a target loss function. That is, optionally, in some embodiments of this application, the training steps of the image recognition model include: Obtain a sample dataset, which includes a normal sample set and an abnormal sample set. The sample acoustic imaging cloud map corresponding to the abnormal sample set is marked with sample abnormality type information, sample abnormality location information and sample optimization indication information. The sample dataset is divided into a training set, a validation set, and a test set; Construct a target loss function, which includes a cross-entropy loss function and / or an intersection-union ratio (IURR) loss function. The cross-entropy loss function is used to characterize the coordinate difference between the predicted anomaly location information and the sample anomaly location information. The IURR loss function is used to characterize the overlap difference between the predicted anomaly location information and the sample anomaly location information. The image recognition model is obtained by training the initial recognition model based on the training set, the validation set, the test set, and the target loss function.

[0079] In this embodiment, the sample dataset is a collection of data used to train, validate, and test the image recognition model. For example, the sample dataset may include acoustic imaging cloud images of samples collected when multiple target device samples play target audio, as well as annotation information corresponding to the sample acoustic imaging cloud images.

[0080] The normal sample set is used to indicate the set of samples that do not have preset anomalies. For example, the normal sample set may include acoustic imaging cloud maps of target equipment samples that are free of vibration, leaks, or whose acoustic conditions meet the standards.

[0081] The abnormal sample set is used to indicate a set of samples with preset anomalies. For example, the abnormal sample set may include acoustic imaging cloud maps of target equipment samples that have been confirmed by manual listening to have vibrations, air leaks, wind noise, or other anomalies.

[0082] The sample anomaly type information is used to indicate the actual anomaly type label corresponding to the acoustic imaging cloud map of the sample in the anomaly sample set. For example, the sample anomaly type information can be labels such as vibration, air leakage, and wind noise marked by engineers.

[0083] Among them, the sample anomaly location information is used to indicate the true location of anomalies in the sample set on the sample acoustic imaging cloud map. For example, the sample anomaly location information can be manually annotated anomaly point coordinates, anomaly region bounding boxes, anomaly region masks, or the partitions where the anomaly region is located.

[0084] Among them, the sample optimization indication information is used to indicate the optimization processing labels corresponding to abnormal samples. For example, the sample optimization indication information can be the improvement measures determined by engineers based on the type and location of the sample anomaly, such as tightening screws, adding EVA stickers, adjusting sealant, and checking structural joints.

[0085] The sample dataset can be randomly divided into a training set, a validation set, and a test set according to a preset ratio. The training set is a subset of data used to update the initial recognition model parameters; the validation set is a subset of data used to evaluate the model training process and help determine whether overfitting has occurred; and the test set is a subset of data used to evaluate the performance of the model after training.

[0086] The target loss function is a function used to measure the difference between the predicted result and the sample labeling result when training the initial recognition model. For example, the target loss function can include the cross-entropy loss function, the cross-union ratio loss function, the localization loss function, and the optimized indication information generation loss function.

[0087] The cross-entropy loss function is used to indicate the difference between the predicted label distribution and the true label distribution. In this embodiment, when the anomaly location information is represented in the form of discrete coordinates, grid positions, partition categories, or coordinate intervals, the cross-entropy loss function can be used to characterize the coordinate difference between the predicted anomaly location information and the sample anomaly location information. For example, an acoustic imaging cloud map can be divided into multiple location grids, the model predicts the probability distribution of the grid to which the anomaly location belongs, and the cross-entropy loss function is used to constrain the predicted grid output by the model to be consistent with the labeled grid.

[0088] The Cross-Union Ratio (CUI) loss function is used to indicate the difference in the degree of overlap between the predicted anomalous region and the sample anomalous region. For example, when the model outputs the bounding box of the anomalous sound source or the anomalous region mask, the intersection area and union area between the predicted anomalous region and the labeled anomalous region can be calculated, and the loss can be determined based on the CUI. The CUI loss can comprehensively reflect the positional deviation, region size, and aspect ratio differences of the anomalous region.

[0089] The initial recognition model refers to a model that has not yet completed training or has not yet reached the preset performance requirements. For example, the initial recognition model can be a neural network model after setting the initial hyperparameters.

[0090] In summary, during training, a large number of product examples can be acquired as a sample dataset, and acoustic imaging cloud images of the samples when the target audio is played are captured using an acoustic camera. For anomalous samples, engineers or annotators can annotate the sample acoustic imaging cloud images with information on the anomaly type, location, and optimization instructions. Then, the sample dataset is divided into training, validation, and test sets. The parameters of the initial recognition model are updated based on the training set and the target loss function. The validation set is used to determine whether the model is overfitting or needs retraining, and the test set is used to evaluate whether the model meets the preset performance requirements. After training, an image recognition model suitable for actual production lines or R&D testing scenarios can be obtained.

[0091] Furthermore, to improve model robustness, data augmentation can be performed on the acoustic imaging cloud maps of the samples. For example, the acoustic imaging cloud maps of the samples can be processed by flipping, scaling, rotating, deforming, color perturbation, and noise superposition, so that the model can learn the correlation between spatial pixels in the acoustic imaging cloud maps, thereby improving the model's adaptability to different product postures, different sound field distributions, and different degrees of anomalies.

[0092] In summary, this application embodiment acquires acoustic imaging cloud images when the target device plays target audio, and performs feature extraction and anomaly type identification based on the acoustic imaging cloud images. Compared with the method of simply relying on manual listening, this application embodiment can improve the objectivity and consistency of anomaly type identification results, reduce the impact of differences in human experience on detection results, and improve the efficiency of target device anomaly detection.

[0093] Furthermore, by further determining the anomaly location information, target component information, and optimization instruction information, the embodiments of this application can associate the anomaly type with the location of the anomaly and the specific component of the target equipment. Therefore, it can not only identify whether the target equipment has anomalies such as vibration or air leakage and the type of the anomaly, but also provide more specific target components and optimization directions for anomaly handling, which helps reduce the time engineers spend repeatedly listening, disassembling, and troubleshooting.

[0094] In this application, by mapping cloud map partitions to device component information one-to-one, or by registering the coordinates of device images and acoustic imaging cloud maps to determine target component information, the embodiments of this application can achieve the correspondence between abnormal locations and target components at a lower cost in fixed product scenarios, and can also improve the automation level and applicability of target component determination in different product or different structural layout scenarios.

[0095] By correcting the anomaly type information based on the target component information, this embodiment of the application can further determine the anomaly type identification result by combining the actual structural location of the target device, making the corrected anomaly type information more consistent with the actual cause of the anomaly, thereby improving the accuracy of the optimization indication information. For example, for the same anomaly of air leakage, different optimization indication information can be provided depending on whether it occurs at a screw hole or at a structural joint.

[0096] In this embodiment, through the cooperation of the image recognition model and the first feature extraction module, the first recognition module, the second feature extraction module, the second recognition module, the positioning module, the optimization module, and the output module, the acoustic features in the acoustic imaging cloud map and the spatial structural features in the device image can be extracted respectively, and the abnormal position can be matched with the position of the device component, thereby realizing the automated processing of abnormal type identification, abnormal position determination, target component positioning, and optimization indication information output.

[0097] In this embodiment, by constructing a sample dataset containing both normal and abnormal samples during training, and annotating the abnormality type, location, and optimization indication information on the acoustic imaging cloud map corresponding to the abnormal samples, the image recognition model can learn the correlation between abnormality type, location, and optimization indication information. Training the initial recognition model using the cross-entropy loss function and / or the cross-union ratio loss function can constrain the differences in discrete location predictions and the differences in abnormal region overlap, thereby improving the accuracy of abnormal location prediction and abnormal region detection.

[0098] To facilitate better implementation of the anomaly identification method of this application, this application also provides an anomaly identification device based on the above-described anomaly identification method. The meanings of the terms used are the same as in the anomaly identification method described above, and specific implementation details can be found in the descriptions of the method embodiments.

[0099] Please see Figure 3 , Figure 3 This is a schematic diagram of the anomaly detection device provided in an embodiment of this application. The anomaly detection device is applied to a terminal device, and the anomaly detection device can specifically be as follows: The acquisition module 301 is used to acquire acoustic imaging cloud images collected when the target device plays the target audio; Extraction module 302 is used to extract acoustic feature information from the acoustic imaging cloud map; The identification module 303 is used to identify the abnormality type information of the target device based on the acoustic feature information.

[0100] Optionally, in some embodiments of this application, the device further includes: Anomaly location information is determined on the acoustic imaging cloud map based on the acoustic feature information, wherein the anomaly location information is the location of the anomaly corresponding to the anomaly type information on the acoustic imaging cloud map; Based on the abnormal location information, target component information for the target device is determined, and the target component information is used to indicate the device components on the target device; The optimization instruction information corresponding to the anomaly type information is determined based on the target component information.

[0101] Optionally, in some embodiments of this application, locating the target component information on the target device based on the abnormal location information includes: The abnormal location information is determined to be located in the cloud map partition on the acoustic imaging cloud map, and the device component information corresponding to the cloud map partition is determined as the target component information, wherein the cloud map partition and the device component information of the target device correspond one-to-one; And / or, Acquire a device image when the target device plays the target audio, register the coordinates of the device image with the acoustic imaging cloud map, and determine the target component information based on the abnormal location information and the device image.

[0102] Optionally, in some embodiments of this application, determining the optimization indication information corresponding to the anomaly type information based on the target component information includes: The optimization instruction information is determined based on the target component information and the anomaly type information; And / or, The abnormality type information is corrected based on the target component information to obtain corrected abnormality type information, and the optimization indication information is determined based on the target component information and the corrected abnormality type information.

[0103] Optionally, in some embodiments of this application, the step of extracting acoustic feature information from the acoustic imaging cloud image includes: The acoustic feature information is obtained by extracting the feature information of the acoustic imaging cloud map through an image recognition model; The device also includes: The image recognition model identifies the anomaly type information corresponding to the acoustic feature information and determines the anomaly location information corresponding to the anomaly type information. The target component information is determined by the image recognition model based on the abnormal location information and the device image; The image recognition model corrects the anomaly type information based on the target component information to obtain the corrected anomaly type information; The optimization indication information is determined by the image recognition model based on the target component information, the anomaly type information, and / or the corrected anomaly type information.

[0104] Optionally, in some embodiments of this application, the image recognition model includes a first feature extraction module, a first recognition module, a second feature extraction module, a second recognition module, a localization module, an optimization module, and an output module; The input terminal of the first feature extraction module is configured to receive the acoustic imaging cloud map, the output terminal of the first feature extraction module is connected to the input terminal of the first recognition module, and the output terminal of the first recognition module is connected to the input terminal of the positioning module. The input terminal of the second feature extraction module is configured to receive the device image, the output terminal of the second feature extraction module is connected to the input terminal of the second recognition module, and the output terminal of the second recognition module is connected to the input terminal of the positioning module. The output of the positioning module is connected to the input of the optimization module, and the output of the optimization module is connected to the input of the output module. The output of the output module is configured to output the target component information, the anomaly type information, the corrected anomaly type information, or the optimization indication information. The first feature extraction module is used to extract features from the acoustic imaging cloud map to obtain acoustic feature information; The first identification module is used to identify the acoustic feature information to obtain the anomaly type information and the anomaly location information; The second feature extraction module is used to extract features from the device image to obtain spatial structure feature information of the device; The second identification module is used to identify the spatial structure feature information of the device to obtain the spatial position information of each device component of the target device; The positioning module is used to locate the target device component from each of the device components of the target device according to the abnormal location information and the spatial location information, and to obtain the target component information; The optimization module is used to correct the anomaly type information based on the target component information to obtain the corrected anomaly type information; The output module is used to determine the optimization indication information based on the target component information, the anomaly type information and / or the corrected anomaly type information, and output the target component information, the anomaly type information, the corrected anomaly type information or the optimization indication information.

[0105] Optionally, in some embodiments of this application, the training steps of the image recognition model include: Obtain a sample dataset, which includes a normal sample set and an abnormal sample set. The sample acoustic imaging cloud map corresponding to the abnormal sample set is marked with sample abnormality type information, sample abnormality location information and sample optimization indication information. The sample dataset is divided into a training set, a validation set, and a test set; Construct a target loss function, which includes a cross-entropy loss function and / or an intersection-union ratio (IURR) loss function. The cross-entropy loss function is used to characterize the coordinate difference between the predicted anomaly location information and the sample anomaly location information. The IURR loss function is used to characterize the overlap difference between the predicted anomaly location information and the sample anomaly location information. The image recognition model is obtained by training the initial recognition model based on the training set, the validation set, the test set, and the target loss function.

[0106] In this embodiment, the acquisition module 301 acquires an acoustic imaging cloud image collected when the target device plays target audio, the extraction module 302 extracts features from the acoustic imaging cloud image to obtain acoustic feature information, and the identification module 303 identifies the abnormality type information of the target device based on the acoustic feature information.

[0107] In summary, this application embodiment acquires acoustic imaging cloud images when the target device plays target audio, and performs feature extraction and anomaly type identification based on the acoustic imaging cloud images. Compared with the method of simply relying on manual listening, this application embodiment can improve the objectivity and consistency of anomaly type identification results, reduce the impact of differences in human experience on detection results, and improve the efficiency of target device anomaly detection.

[0108] In addition, this application also provides an electronic device, such as Figure 4 As shown, it illustrates a structural schematic diagram of the electronic device provided in an embodiment of this application. Specifically: The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will understand that... Figure 4The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 401 is the control center of the electronic device. It connects various parts of the electronic device via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 402, and by calling data stored in the memory 402, it performs various functions and processes data, thereby providing overall monitoring of the electronic device. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 401.

[0109] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0110] The electronic device also includes a power supply 403 that supplies power to the various components. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 403 may also include one or more DC or AC power supplies, recharging systems, power equipment debugging circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0111] The electronic device may also include an input unit 404, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0112] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 runs the applications stored in the memory 402, thereby implementing the steps in any of the anomaly identification methods provided in the embodiments of this application.

[0113] This application embodiment acquires an acoustic imaging cloud map collected when the target device plays target audio; performs feature extraction on the acoustic imaging cloud map to obtain acoustic feature information, and identifies the abnormal type information of the target device based on the high acoustic feature information.

[0114] In this embodiment, the acoustic imaging cloud map of the target device when playing target audio is collected, and feature extraction and anomaly type identification are performed based on the acoustic imaging cloud map. Compared with the method of simply relying on human listening, this embodiment can improve the objectivity and consistency of anomaly type identification results, reduce the impact of human experience differences on detection results, and improve the efficiency of target device anomaly detection.

[0115] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0116] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0117] Therefore, this application provides a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps in any of the anomaly identification methods provided in this application.

[0118] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0119] The computer-readable storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0120] Since the instructions stored in the computer-readable storage medium can execute the steps of any of the anomaly identification methods provided in this application, the beneficial effects that any of the anomaly identification methods provided in this application can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.

[0121] The above provides a detailed description of an anomaly identification method, apparatus, electronic device, and computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0122] It is understood that in the specific embodiments of this application, data such as acoustic imaging cloud map and device image of the target device when the target device plays the target audio are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. An anomaly identification method, characterized in that, The method includes: Acoustic imaging cloud images were captured when the target device played the target audio. Acoustic feature information is obtained by extracting features from the acoustic imaging cloud map; The abnormality type information of the target device is identified based on the acoustic feature information.

2. The anomaly identification method according to claim 1, characterized in that, The method further includes: Anomaly location information is determined on the acoustic imaging cloud map based on the acoustic feature information, wherein the anomaly location information is the location of the anomaly corresponding to the anomaly type information on the acoustic imaging cloud map; Based on the abnormal location information, target component information for the target device is determined, and the target component information is used to indicate the device components on the target device; The optimization instruction information corresponding to the anomaly type information is determined based on the target component information.

3. The anomaly identification method according to claim 2, characterized in that, The step of locating the target component information on the target device based on the abnormal location information includes: The abnormal location information is determined to be located in the cloud map partition on the acoustic imaging cloud map, and the device component information corresponding to the cloud map partition is determined as the target component information, wherein the cloud map partition and the device component information of the target device correspond one-to-one; And / or, Acquire a device image when the target device plays the target audio, register the coordinates of the device image with the acoustic imaging cloud map, and determine the target component information based on the abnormal location information and the device image.

4. The anomaly identification method according to claim 3, characterized in that, The step of determining the optimization indication information corresponding to the anomaly type information based on the target component information includes: The optimization instruction information is determined based on the target component information and the anomaly type information; And / or, The abnormality type information is corrected based on the target component information to obtain corrected abnormality type information, and the optimization indication information is determined based on the target component information and the corrected abnormality type information.

5. The anomaly identification method according to claim 4, characterized in that, The process of extracting acoustic feature information from the acoustic imaging cloud image includes: The acoustic feature information is obtained by extracting the feature information of the acoustic imaging cloud map through an image recognition model; The method further includes: The image recognition model identifies the anomaly type information corresponding to the acoustic feature information and determines the anomaly location information corresponding to the anomaly type information. The target component information is determined by the image recognition model based on the abnormal location information and the device image; The image recognition model corrects the anomaly type information based on the target component information to obtain the corrected anomaly type information; The optimization indication information is determined by the image recognition model based on the target component information, the anomaly type information, and / or the corrected anomaly type information.

6. The anomaly identification method according to claim 5, characterized in that, The image recognition model includes a first feature extraction module, a first recognition module, a second feature extraction module, a second recognition module, a localization module, an optimization module, and an output module. The input terminal of the first feature extraction module is configured to receive the acoustic imaging cloud map, the output terminal of the first feature extraction module is connected to the input terminal of the first recognition module, and the output terminal of the first recognition module is connected to the input terminal of the positioning module. The input terminal of the second feature extraction module is configured to receive the device image, the output terminal of the second feature extraction module is connected to the input terminal of the second recognition module, and the output terminal of the second recognition module is connected to the input terminal of the positioning module. The output of the positioning module is connected to the input of the optimization module, and the output of the optimization module is connected to the input of the output module. The output of the output module is configured to output the target component information, the anomaly type information, the corrected anomaly type information, or the optimization indication information. The first feature extraction module is used to extract features from the acoustic imaging cloud map to obtain acoustic feature information; The first identification module is used to identify the acoustic feature information to obtain the anomaly type information and the anomaly location information; The second feature extraction module is used to extract features from the device image to obtain spatial structure feature information of the device; The second identification module is used to identify the spatial structure feature information of the device to obtain the spatial position information of each device component of the target device; The positioning module is used to locate the target device component from each of the device components of the target device according to the abnormal location information and the spatial location information, and to obtain the target component information; The optimization module is used to correct the anomaly type information according to the target component information to obtain the corrected anomaly type information, and to determine the optimization instruction information according to the target component information, the anomaly type information and / or the corrected anomaly type information; The output module is used to output the target component information, the anomaly type information, the corrected anomaly type information, or the optimization indication information.

7. The anomaly identification method according to claim 5, characterized in that, The training steps of the image recognition model include: Obtain a sample dataset, which includes a normal sample set and an abnormal sample set. The sample acoustic imaging cloud map corresponding to the abnormal sample set is marked with sample abnormality type information, sample abnormality location information and sample optimization indication information. The sample dataset is divided into a training set, a validation set, and a test set; Construct a target loss function, which includes a cross-entropy loss function and / or an intersection-union ratio (IURR) loss function. The cross-entropy loss function is used to characterize the coordinate difference between the predicted anomaly location information and the sample anomaly location information. The IURR loss function is used to characterize the overlap difference between the predicted anomaly location information and the sample anomaly location information. The image recognition model is obtained by training the initial recognition model based on the training set, the validation set, the test set, and the target loss function.

8. An anomaly detection device, characterized in that, The device includes: The acquisition module is used to acquire acoustic imaging cloud images collected when the target device plays the target audio; The extraction module is used to extract acoustic feature information from the acoustic imaging cloud map; The identification module is used to identify the abnormality type information of the target device based on the acoustic feature information.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the anomaly identification method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the anomaly identification method as described in any one of claims 1-7.

Images