Method and device for identifying defects of a motor vehicle, equipment and medium

Abstract

The high-speed train defect identification method and device, equipment and medium provided by the application relate to the technical field of computer vision. In the application, firstly, global scene analysis is performed on a target high-speed train image to obtain image scene description data; secondly, based on the image scene description data, local anomaly analysis is performed on the target high-speed train image to obtain a candidate abnormal region set; then, anomaly semantic verification is performed on each candidate abnormal region in the candidate abnormal region set to obtain a defect identification result of each candidate abnormal region; finally, based on the defect identification result of each candidate abnormal region, a target abnormal region is determined in the candidate abnormal region set, and spatial positioning processing is performed on the target abnormal region to obtain a spatial positioning result. Based on the above, the problem that the reliability of defect identification is relatively low in the prior art can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and more specifically, to a method, apparatus, device, and medium for identifying defects in high-speed trains. Background Technology

[0002] Visual-language multimodal pre-trained models have achieved powerful zero-shot image classification capabilities through contrastive learning on large-scale internet image-text pairs. However, directly applying them to industrial defect detection has significant limitations: the model is trained on natural images, which differ from the distribution of railway industrial images (with specific textures, structures, and lighting). Furthermore, due to the scarcity of samples of severe and novel defects in the relevant field, it is difficult to collect such samples. This leads to feature alignment failure and high false positive and false negative rates. In other words, the reliability of defect identification is relatively low. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a method, apparatus, device and medium for identifying defects in high-speed trains, so as to improve the problem of relatively low reliability of defect identification in the prior art.

[0004] To achieve the above objectives, this application adopts the following technical solution: A method for identifying defects in high-speed trains includes: Global scene analysis is performed on the target train image to obtain image scene description data; Based on the image scene description data, local anomaly analysis is performed on the target train image to obtain a set of candidate anomaly regions. Each candidate anomaly region in the set of candidate anomaly regions refers to an area in the analyzed target train image that is abnormal. Anomaly semantic verification is performed on each candidate anomaly region in the candidate anomaly region set to obtain a defect identification result for each candidate anomaly region, wherein the defect identification result is used to reflect at least one of the defect type, confidence level, and defect discrimination criteria; Based on the defect identification result of each candidate abnormal region, a target abnormal region is determined in the set of candidate abnormal regions, and spatial positioning processing is performed on the target abnormal region to obtain a spatial positioning result. The target abnormal region refers to a candidate abnormal region whose corresponding defect identification result satisfies the pre-configured defect-related conditions, and the spatial positioning result is used to reflect the positioning information of the defect existing in the target abnormal region.

[0005] In a preferred embodiment of this application, in the above-mentioned high-speed train defect identification method, the step of performing anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set to obtain the defect identification result for each candidate anomaly region includes: Context-aware analysis is performed on the target train image to obtain target context data, and based on the target context data, dynamic prompt words for defect identification are determined; Based on the dynamic prompts for defect identification, each candidate abnormal region in the candidate abnormal region set is subjected to abnormal semantic verification to obtain the defect identification result for each candidate abnormal region.

[0006] In a preferred embodiment of this application, the steps of performing context-aware analysis on the target train image to obtain target context data, and determining dynamic prompts for defect identification based on the target context data, include: Context-aware analysis is performed on the target train image to obtain target context data; In a pre-determined fault semantic definition library, target description templates with relevant relationships to the target context data are identified. Based on the target description templates, dynamic prompts for defect identification are determined. The description templates in the fault semantic definition library include multi-layer description data, which includes at least scene-level description data, component-level description data, and defect-level description data. The scene-level description data reflects the overall context of the detection environment, which includes at least the shooting location, vehicle components, and environmental conditions. The component-level description data reflects the normal state characteristics of each key component of the train, which includes at least the component name and location information, standard appearance characteristics, allowable normal variation range, and relationships and spatial constraints between adjacent components. The defect-level description data reflects the semantic definition of each type of defect, which includes defect type and subclass, multi-angle visual feature description, occurrence pattern and typical location, distinguishing points from similar normal phenomena, and severity grading standards.

[0007] In a preferred embodiment of this application, the step of performing context-aware analysis on the target train image to obtain target context data in the above-described high-speed train defect identification method includes: A first context-aware analysis is performed on the target train image to obtain image content information, wherein the image content information is used to characterize component type, spatial layout and key visual attributes; A second context-aware analysis is performed on the target train image to obtain environmental factor information, wherein the environmental factor information is used to characterize lighting conditions, shooting angle and image sharpness; The target train image is subjected to third context-aware analysis to obtain interference factor information, wherein the interference factor information is used to characterize the degree of influence of stains, reflections, and obstructions on the detection. Based on the image content information, the environmental factor information, and the interference factor information, the target context data is determined.

[0008] In a preferred embodiment of this application, the step of performing global scene analysis on the target train image to obtain image scene description data in the above-mentioned high-speed train defect identification method includes: Obtain scene parsing prompt data configured for the target train image; Based on the scene analysis prompt data, global scene analysis is performed on the target train image to obtain image scene description data, wherein the image scene description data includes at least a component list, relative position and environmental features of each component of the train.

[0009] In a preferred embodiment of this application, in the above-mentioned high-speed train defect identification method, the step of performing local anomaly analysis on the target high-speed train image based on the image scene description data to obtain a set of candidate anomaly regions includes: Obtain anomaly detection alert data configured for the target train image; Based on the anomaly detection prompt data and the image scene description data, local anomaly analysis is performed on the target train image to obtain a set of candidate anomaly regions.

[0010] In a preferred embodiment of this application, in the above-described high-speed train defect identification method, the steps of determining a target abnormal region from the set of candidate abnormal regions based on the defect identification result of each candidate abnormal region, and performing spatial localization processing on the target abnormal region to obtain a spatial localization result, include: Based on the defect identification results of each candidate abnormal region, the target abnormal region is determined from the set of candidate abnormal regions; Obtain a precise positioning instruction configured for the target abnormal area, and perform spatial positioning processing on the target abnormal area based on the precise positioning instruction to obtain a spatial positioning result. The precise positioning instruction includes generating a minimum bounding rectangle for linear defects, determining the main influence range for diffuse defects, and merging and segmenting regions for multi-point defects. The linear defects include scratches, and the diffuse defects include oil stains.

[0011] This application also provides a vehicle defect identification device, including: The global scene analysis module is used to perform global scene analysis on the target train image to obtain image scene description data; The local anomaly analysis module is used to perform local anomaly analysis on the target train image based on the image scene description data to obtain a set of candidate anomaly regions, wherein each candidate anomaly region in the set of candidate anomaly regions refers to the region in the analyzed target train image that is abnormal. An anomaly semantic verification module is used to perform anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set to obtain a defect identification result for each candidate anomaly region, wherein the defect identification result is used to reflect at least one of the defect type, confidence level and defect discrimination criteria. The spatial positioning processing module is used to determine the target abnormal region in the set of candidate abnormal regions based on the defect identification result of each candidate abnormal region, and to perform spatial positioning processing on the target abnormal region to obtain a spatial positioning result. The target abnormal region refers to the candidate abnormal region whose corresponding defect identification result satisfies the pre-configured defect-related conditions. The spatial positioning result is used to reflect the positioning information of the defect existing in the target abnormal region.

[0012] Based on the above, this application also provides an electronic device, including: Memory, used to store computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the above-described method for identifying defects in high-speed trains.

[0013] Based on the above, this application also provides a computer-readable storage medium storing a computer program that executes the various steps of the above-described method for identifying defects in high-speed trains when the computer program is run.

[0014] The method, apparatus, equipment, and medium for identifying defects in high-speed trains provided in this application firstly perform global scene analysis on the target high-speed train image to obtain image scene description data; secondly, based on the image scene description data, perform local anomaly analysis on the target high-speed train image to obtain a set of candidate anomaly regions; then, perform anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set to obtain a defect identification result for each candidate anomaly region; finally, based on the defect identification result for each candidate anomaly region, determine the target anomaly region in the candidate anomaly region set, and perform spatial localization processing on the target anomaly region to obtain a spatial localization result. Based on the above, because global scene analysis, local anomaly analysis, anomaly semantic verification, and spatial localization processing are performed sequentially, defect identification and localization are multi-stage and progressive, achieving sufficient guidance for defect identification and localization, decomposing complex defect detection tasks, ensuring the accuracy of defect identification and localization, and thus improving the problem of relatively low reliability of defect identification in existing technologies. Attached Figure Description

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0016] Figure 1 A structural block diagram of an electronic device provided in an embodiment of this application.

[0017] Figure 2 This is a flowchart illustrating the method for identifying defects in high-speed trains provided in an embodiment of this application.

[0018] Figure 3 This is a block diagram of a train defect identification device provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0021] like Figure 1As shown in the illustration, this application provides an electronic device. The electronic device may include a memory, a processor, and a vehicle defect identification device.

[0022] In detail, the memory and the processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, the memory and the processor can be electrically connected via one or more communication buses or signal lines. The train defect identification device includes at least one software functional module stored in the memory in the form of software or firmware. The processor is used to execute executable computer programs stored in the memory, such as the software functional modules and computer programs included in the train defect identification device, to implement the train defect identification method provided in the embodiments of this application.

[0023] Optionally, the memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0024] Furthermore, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0025] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown may include, for example, a communication unit for exchanging information with other devices (such as image acquisition devices).

[0026] Combination Figure 2This application also provides a method for identifying defects in high-speed trains that can be applied to the aforementioned electronic equipment. The method steps defined in the process of the high-speed train defect identification method can be implemented by the electronic equipment. The following will describe... Figure 2 The specific process shown will be explained in detail.

[0027] Step S110: Perform global scene analysis on the target train image to obtain image scene description data.

[0028] In this embodiment of the application, the electronic device can perform global scene analysis on the target train image to obtain image scene description data. For example, the image scene description data can be used to reflect information such as components present in the target train image.

[0029] Step S120: Based on the image scene description data, perform local anomaly analysis on the target train image to obtain a set of candidate anomaly regions.

[0030] In this embodiment, after obtaining the image scene description data, the electronic device can perform local anomaly analysis on the target train image based on the image scene description data to obtain a set of candidate anomaly regions. Each candidate anomaly region in the set refers to a region in the analyzed target train image that exhibits anomalies. In other words, preliminary analysis and identification are performed first to identify potential anomaly regions as candidates, thus obtaining the corresponding candidate anomaly regions.

[0031] Step S130: Perform anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set to obtain the defect identification result of each candidate anomaly region.

[0032] In this embodiment, after obtaining the set of candidate abnormal regions, the electronic device can perform anomaly semantic verification on each candidate abnormal region in the set to obtain a defect identification result for each candidate abnormal region. The defect identification result reflects at least one of the following: defect type, confidence level, and defect discrimination criteria. That is, for suspected abnormal regions, i.e., regions that may contain defects, further analysis and identification are performed to determine information such as the defect type, confidence level, and defect discrimination criteria.

[0033] Step S140: Based on the defect identification result of each candidate abnormal region, determine the target abnormal region in the set of candidate abnormal regions, and perform spatial positioning processing on the target abnormal region to obtain the spatial positioning result.

[0034] In this embodiment, after obtaining the defect identification result, the electronic device can determine the target abnormal region from the set of candidate abnormal regions based on the defect identification result of each candidate abnormal region, and perform spatial positioning processing on the target abnormal region to obtain a spatial positioning result. The target abnormal region refers to a candidate abnormal region whose corresponding defect identification result satisfies pre-configured defect-related conditions. The spatial positioning result reflects the location information of the defect present in the target abnormal region. In other words, based on the defect identification results of each suspected abnormal region, further analysis and confirmation can be performed to obtain the target abnormal region, i.e., to determine the region where an anomaly exists. Then, spatial positioning processing can be performed on the target abnormal region, making the obtained spatial positioning result more reliable.

[0035] Based on the above, since global scene analysis, local anomaly analysis, anomaly semantic verification, and spatial localization processing are performed sequentially, the identification and localization of defects are multi-stage and progressive. This fully guides the identification and localization of defects, decomposes complex defect detection tasks, ensures the accuracy of defect identification and localization, and improves the problem of relatively low reliability of defect identification in existing technologies.

[0036] Firstly, regarding step S110, it should be noted that the specific method for performing global scene analysis on the target train image is not limited and can be selected according to actual needs.

[0037] For example, in an alternative implementation, in order to improve the reliability of global scene parsing and enable the obtained image scene description data to effectively represent the scene-related information or features in the target train image, the above step S110 may further include steps S111 and S112, wherein the specific contents of each step are as follows.

[0038] Step S111: Obtain scene parsing prompt data configured for the target train image.

[0039] In this embodiment of the application, scene analysis prompt data configured for the target train image can be obtained, which can guide the effective execution of subsequent global scene analysis. For example, it can be "generating a list of each component, the relative position of each component, and the environmental conditions".

[0040] Step S112: Based on the scene parsing prompt data, perform global scene parsing on the target train image to obtain image scene description data.

[0041] In this embodiment, after obtaining the scene analysis prompt data, global scene analysis can be performed on the target train image based on the scene analysis prompt data to obtain image scene description data. The image scene description data includes at least a component list, relative positions, and environmental features (such as lighting conditions) of each component of the train. It should be noted that global scene analysis can be implemented using existing visual language models. For example, after semantically encoding the target train image to obtain corresponding semantic information, it can be fused with the semantic information of the scene analysis prompt data (e.g., through splicing, overlay, or other processing methods). Then, the fused semantic information can be decoded and output to obtain the image scene description data.

[0042] Secondly, regarding step S120, it should be noted that the specific method for performing local anomaly analysis on the target train image is not limited and can be selected according to actual needs.

[0043] For example, in an alternative implementation, in order to improve the accuracy of local anomaly analysis and make the reliability of the obtained candidate anomaly region set relatively higher, the above step S120 may further include steps S121 and S122, wherein the specific contents of each step are as follows.

[0044] Step S121: Obtain anomaly detection prompt data configured for the target train image.

[0045] In this embodiment of the application, abnormal detection prompt data configured for the target train image can be obtained, such as "identifying areas that look abnormal, each area containing location information and a preliminary abnormal description, the preliminary abnormal description including...".

[0046] Step S122: Based on the anomaly detection prompt data and the image scene description data, perform local anomaly analysis on the target train image to obtain a set of candidate anomaly regions.

[0047] In this embodiment, after obtaining the anomaly detection prompt data, local anomaly analysis can be performed on the target train image based on the anomaly detection prompt data and the image scene description data to obtain a set of candidate anomaly regions. That is, the anomaly detection prompt data, the image scene description data, and the target train image can be semantically encoded using a visual language model. Then, the three encoded semantic information can be fused. Finally, the fused semantic information can be semantically decoded to form a set of candidate anomaly regions. It should be noted that each subsequent anomaly region can also generate a corresponding anomaly description, such as the anomaly type.

[0048] Thirdly, regarding step S130, it should be noted that the specific method for performing anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set is not limited and can be selected according to actual needs.

[0049] For example, in an alternative implementation, in order to improve the accuracy of the abnormal semantic verification and make the reliability of the obtained defect identification results higher, the above step S130 may further include steps S131 and S132, wherein the specific contents of each step are as follows.

[0050] Step S131: Perform context-aware analysis on the target train image to obtain target context data, and determine dynamic prompt words for defect identification based on the target context data.

[0051] In this embodiment, context-aware analysis can be performed on the target train image to obtain target context data, and based on the target context data, dynamic defect identification prompts can be determined. In other words, information potentially related to defect identification in the target train image can be analyzed, allowing dynamic defect identification prompts to be generated based on this information. That is, the prompts are dynamically generated based on information in the image, rather than being fixed. This improves the accuracy of subsequent anomaly semantic verification based on the dynamic defect identification prompts.

[0052] Step S132: Based on the defect identification dynamic prompt words, perform abnormal semantic verification on each candidate abnormal region in the candidate abnormal region set to obtain the defect identification result of each candidate abnormal region.

[0053] In this embodiment, after obtaining the dynamic prompt for defect identification, anomaly semantic verification can be performed on each candidate anomaly region in the candidate anomaly region set based on the dynamic prompt, thereby obtaining the defect identification result for each candidate anomaly region. For example, a visual language model can be used to semantically encode the image information corresponding to the dynamic prompt for defect identification and the candidate anomaly region, respectively. Then, the two encoded semantic information can be fused, and finally, the fused semantic information can be decoded and output to obtain the defect identification result corresponding to the candidate anomaly region. It should be noted that a cross-attention mechanism can be used to achieve cross-modal fusion of text and images to ensure the accuracy of semantic fusion.

[0054] It is understood that the specific method of determining the defect identification dynamic prompt words in step S131 above is not limited. For example, in an alternative implementation, in order to ensure the reliability of the determined defect identification dynamic prompt words, step S131 above may further include steps S131a and S131b, wherein the specific contents of each step are as follows.

[0055] Step S131a: Perform context-aware analysis on the target train image to obtain target context data.

[0056] In this embodiment of the application, context-aware analysis can be performed on the target train image to obtain target context data. That is, context analysis can be performed on the target train image to obtain target context data that can fully characterize the context of the target train image.

[0057] Step S131b: In a pre-determined fault semantic definition library, a target description template that has a relevant relationship with the target context data is determined, and based on the target description template, a dynamic prompt word for defect identification is determined.

[0058] In this embodiment, after obtaining the target context data, a target description template with a correlation to the target context data (such as the template with the highest matching degree with the target context data, the matching degree calculation can refer to relevant prior art) can be determined from a pre-determined fault semantic definition library. Based on the target description template, dynamic prompts for defect identification are determined. The description templates in the fault semantic definition library include multi-layered description data, which includes at least scene-level description data, component-level description data, and defect-level description data. The scene-level description data reflects the overall context of the detection environment, including at least the shooting location, vehicle components, and environmental conditions. The component-level description data reflects the normal state characteristics of each key component of the train, including at least the component name and location information, standard appearance features, allowable normal variation range, and relationships and spatial constraints between adjacent components. The defect-level description data reflects the semantic definition of each type of defect, including defect type and subclass, multi-angle visual feature description, occurrence pattern and typical location, distinguishing points from similar normal phenomena, and severity grading standards. For example, the dynamic prompts for defect identification can include basic prompts, supplementary explanations, and interference elimination. For instance, if "component = skirt panel, lighting conditions = overexposure", then the dynamic prompts for defect identification could be: basic prompt = "detect abnormal damage, scratches, and foreign objects on the skirt panel surface", supplementary explanation = "Note: Under overexposure conditions, damaged areas may appear as reduced brightness. Please analyze the uniformity of the vehicle texture", and interference elimination = "eliminate brightness changes caused by reflections and interference from inherent labels on the vehicle body".

[0059] It is understood that the specific method of performing context-aware analysis on the target train image in step S131a above is not limited. For example, in an alternative implementation, in order to ensure that the obtained target context data has high reliability and that a reliable target description template can be determined based on the target context data, step S131a above may further include steps a1, a2, a3 and a4, wherein the specific contents of each step are as follows.

[0060] Step a1: Perform a first context-aware analysis on the target train image to obtain image content information.

[0061] In this embodiment of the application, a first context-aware analysis can be performed on the target train image to obtain image content information. This image content information is used to characterize component type, spatial layout (such as the relative positional relationship between components), and key visual attributes (such as geometric shape and size).

[0062] Step a2: Perform a second context-aware analysis on the target train image to obtain environmental factor information.

[0063] In this embodiment of the application, a second context-aware analysis can also be performed on the target train image to obtain environmental factor information. This environmental factor information is used to characterize lighting conditions (such as front lighting, backlighting, and shadow distribution), shooting angle, and image sharpness.

[0064] Step a3: Perform third context-aware analysis on the target train image to obtain interference factor information.

[0065] In this embodiment of the application, a third context-aware analysis can also be performed on the target train image to obtain interference factor information. This interference factor information is used to characterize the degree of influence of stains, reflections, and obstructions on the detection.

[0066] Step a4: Determine the target context data based on the image content information, the environmental factor information, and the interference factor information.

[0067] In this embodiment, after obtaining the image content information, the environmental factor information, and the interference factor information, target context data can be determined based on these three information. That is, the target context data can include the image content information, the environmental factor information, and the interference factor information. Furthermore, the aforementioned first context-aware analysis, second context-aware analysis, and third context-aware analysis can be implemented using a neural network model. Semantic encoding and semantic decoding can be performed on the target train image to obtain the image content information, the environmental factor information, and the interference factor information, respectively. Alternatively, the target train image can be semantically encoded once, and then the image content information, the environmental factor information, and the interference factor information can be obtained through different semantic decoding methods.

[0068] Fourthly, regarding step S140, it should be noted that the specific method for spatial positioning processing of the target abnormal area is not limited and can be selected according to actual needs.

[0069] For example, in an alternative implementation, in order to improve the accuracy of spatial positioning processing and make the resulting spatial positioning results more reliable, the above-mentioned step S140 may further include steps S141 and S142, wherein the specific contents of each step are as follows.

[0070] Step S141: Based on the defect identification result of each candidate abnormal region, determine the target abnormal region from the set of candidate abnormal regions.

[0071] In this embodiment of the application, a target abnormal region can be determined from the set of candidate abnormal regions based on the defect identification result of each candidate abnormal region. For example, the defect identification result can be used to reflect the type of defect, the confidence level, and the criteria for defect judgment. For instance, each candidate abnormal region that meets the criteria can be determined based on the corresponding confidence level, and then identified as the target abnormal region, such as when the confidence level is greater than a predetermined reference threshold, such as 0.6 or 0.7.

[0072] Step S142: Obtain the precise positioning instruction configured for the target abnormal area, and perform spatial positioning processing on the target abnormal area based on the precise positioning instruction to obtain the spatial positioning result.

[0073] In this embodiment, after obtaining the target anomaly region, a precise positioning instruction configured for the target anomaly region can be acquired. Based on the precise positioning instruction, spatial positioning processing is performed on the target anomaly region to obtain a spatial positioning result. That is, the precise positioning instruction and the target anomaly region (which can be segmented from the target train image) are semantically encoded using a visual language model. Then, the two encoded semantic information are fused. Finally, the fused semantic information is decoded and output to obtain a spatial positioning result (such as pixel-level positioning information of the defect, including bounding box coordinates and spatial coverage). The precise positioning instruction includes generating a minimum bounding rectangle for linear defects, determining the main influence range for diffuse defects, and merging and segmenting regions for multi-point defects. Linear defects include scratches, and diffuse defects include oil stains.

[0074] To facilitate understanding of the above-mentioned EMU defect identification method, this application embodiment also provides a railway EMU defect identification method based on a visual language multimodal model, including the following steps: Step 1: Construct a hierarchical, multi-granular fault semantic definition library A structured semantic knowledge base for defect detection in railway EMUs is established, comprising a three-level description system: Scenario-level description: defining the overall context of the detection environment, including the shooting location (e.g., "bottom," "side"), existing vehicle components (e.g., "gearbox," "traction motor"), and environmental conditions (e.g., "daytime," "nighttime," "rainy weather"); Component-level description: defining in detail the normal state characteristics of each key component of the EMU, including: component name and location information (e.g., "bogie skirt," "non-bogie skirt"), standard appearance characteristics (material, color, texture, reflectivity), and permissible normal variations. Different ranges (e.g., normal wear and tear, permissible surface oxidation), relationships between adjacent components and spatial constraints; Defect-level description: detailed semantic definitions for each type of defect, including: defect type and subclass (e.g., "surface scratches" are subdivided into "shallow scratches" and "deep scratches"), multi-angle visual feature description (shape, size, color, texture changes), occurrence patterns and typical locations (e.g., "oil stains usually spread outward from the seal"), key points for distinguishing from similar normal phenomena (e.g., "distinguishing between reflection and oil stains: reflections have extremely high brightness and change with viewing angle"), severity grading standards (based on factors such as size, depth, and location).

[0075] The aforementioned semantic definition library is stored in an extensible JSON-LD format, supports semantic association queries, and provides rich prior knowledge for subsequent zero-shot inference.

[0076] Step 2: Dynamic Cue Generation Based on Visual Context Analysis Based on the visual features of the input image and the environmental context, generate optimized detection prompts in real time: Context-aware analysis: Uses lightweight vision models to quickly analyze image content, identify main component types, spatial layout and key visual attributes; analyzes environmental factors, including lighting conditions (front lighting, backlighting, shadow distribution), shooting angle and image sharpness; and evaluates the impact of interference factors, such as stains, reflections and occlusions, on detection. Adaptive suggestion construction: Retrieve the most relevant description template to the current context from the fault semantic definition library; dynamically populate template variables according to the specific detection task to generate targeted detection instructions; automatically add auxiliary judgment conditions for difficult scenarios (such as low light, strong reflection); Example suggestion generation logic: If (part = skirt panel and lighting conditions = overexposure): Basic prompt = "Detect abnormal damage, scratches and foreign objects on the skirt panel surface"; Additional notes = "Note: Under overexposure conditions, damaged areas may appear as reduced brightness. Please analyze the uniformity of the vehicle texture"; Interference elimination = "Eliminate brightness changes caused by reflections, as well as interference from inherent labels on the vehicle body"; Step 3: Multi-stage progressive zero-shot inference By employing a phased and progressively refined reasoning strategy, the complex defect detection task is decomposed into four ordered phases: Global Scene Understanding Stage: Input: Complete image of the train + scene analysis prompts; Processing: Visual language model analyzes the overall content of the image, identifies major components and their spatial relationships; Output: Structured scene description, including component list, relative positions, and environmental features; Local anomaly perception stage: Input: Scene understanding results + anomaly detection prompts; Processing: A dual-path anomaly region discovery mechanism is adopted: a) Model self-perception path: guide the model to actively discover regions that "look abnormal"; b) Algorithm-assisted path: generate candidate regions based on unsupervised visual saliency algorithm; Output: a set of candidate anomaly regions, each region containing location information and preliminary anomaly description; Cross-modal semantic verification stage: Input: Images of each candidate region + dynamically generated targeted detection prompts; Processing: For each candidate region, perform multi-angle verification: a) Use a cross-attention mechanism to calculate the fine-grained matching degree between the region's visual features and the defect text description; b) Compare descriptions of multiple related defect types and make a judgment based on the relative matching degree; c) Consider contextual consistency and evaluate the rationality of the defect appearing in the current location; Output: Defect type identification result, matching confidence, and main judgment criteria for each region; Precise Spatial Localization Stage: Input: Defect region that has passed semantic verification + precise localization instructions; Processing: Utilize the model's visual localization capabilities to obtain the precise bounding box of the defect; Special processing: a) For linear defects (such as scratches), generate the minimum bounding rectangle; b) For diffuse defects (such as oil stains), determine the main area of ​​influence; c) For multi-point defects, perform region merging and segmentation; Output: Pixel-level localization information of the defect, including bounding box coordinates and spatial coverage.

[0077] Step 4: Confidence Integration Calibration and Progressive Knowledge Optimization The reliability of the inference results is assessed and the system is optimized: Multi-dimensional confidence calibration: Semantic matching score: the matching score of the cross-modal attention mechanism, reflecting the degree of visual-text alignment; Visual saliency: the prominence of the defect region in the anomaly perception stage; Contextual consistency: the degree of consistency between the defect type and its location and component function; Model self-consistency: the degree of consistency between the results of different inference stages. Integrated confidence score calculation: Final confidence score = w1 × semantic matching degree + w2 × visual saliency + w3 × contextual consistency + w4 × model self-consistency; where w1-w4 are adaptive weights, dynamically adjusted according to defect type and environmental conditions; Uncertainty quantification and result grading: High confidence score results (>0.8): Directly output to guide maintenance decisions; Medium confidence score results (0.6-0.8): Marked for manual review, providing detailed judgment criteria; Low confidence score results (<0.6): Not output temporarily, but recorded for subsequent analysis; Progressive knowledge optimization mechanism: Automatic pseudo-label generation: Collect high-confidence detection results and automatically generate structured labeled data; Feedback learning loop: a) Regularly use pseudo-label data to incrementally train the dedicated detection model; b) Analyze false detection and false negative cases to optimize the accuracy of the fault semantic definition library; c) Update the dynamic prompt generation rules to improve the adaptability to difficult scenarios; Knowledge base version management: Records each optimization and supports the tracking and retrospection of knowledge evolution.

[0078] Step 5: Generation and Visualization of Structured Inspection Report Transform the test results into actionable engineering information: Report Content: Basic Defect Information: Type, Location, Size, Severity; Detection Reliability Indicators: Overall Confidence, Scores for Each Dimension, Uncertainty Range; Judgment Basis Summary: Key Visual Features, Matching Points with Text Description, Excluded Interference Factors; Repair Recommendations: Preliminary Handling Recommendations Based on Defect Type and Severity; Visualization output: Overlay defect location boxes and type labels on the original image; generate interpretable visualizations of the inspection process, showing the areas of focus; provide historical comparison views to show the historical changes in the same area.

[0079] Step 6: System performance monitoring and adaptive adjustment Establish a continuous monitoring mechanism for system operation status: Real-time performance metrics tracking: trends in accuracy, recall, and false positive rate on the validation set; runtime efficiency monitoring: inference time and resource usage at each stage; adaptive adjustment strategy: automatically adding relevant training data when the performance of detecting specific types of defects declines; dynamically adjusting model inference accuracy based on hardware resource conditions (e.g., dynamically adjusting input resolution); automatically updating interference exclusion rules for frequently occurring false positive patterns.

[0080] Detailed Explanation of the Innovations of the Above Methods Innovation Point 1: A hierarchical and structured fault semantic definition library Problem: The simple cue words do not provide enough information.

[0081] Solution: Create a three-level description system: Scene-level: Define the scene to be detected (e.g., "bottom surface of the vehicle body, at the bogie").

[0082] Component level: Describes the normal state of the target component (e.g., "gearbox: intact, dry, and free of deposits on the metal surface").

[0083] Defect Level: Describe in detail the visual characteristics of the defect, the key points for identification, and possible interfering factors (such as "oil stain: dark, wet, with a sense of diffusion at the edges; pay attention to distinguishing it from shadows, the edges of shadows are blurred and not reflective").

[0084] Technical benefits: It provides the model with rich, accurate, and interpretable prior knowledge, forming the cornerstone of zero-shot detection. It digitizes and structures expert experience.

[0085] Innovation Point Two: Dynamic Cue Generation Engine Based on Image Context Problem: Static prompts cannot adapt to changing testing environments (such as different components, lighting, and angles).

[0086] Solution: Design a prompt generation method whose inputs are image metadata (automatically analyzed part type, lighting conditions) and target task (what kind of defect to detect), and whose output is a customized instruction.

[0087] Example: When the system recognizes that the current image is "side of the vehicle" and "strongly reflective", it will automatically add the following to the basic prompt: "Note: Under strong reflective conditions, scratches may appear as bright lines. Please focus on analyzing the texture discontinuity rather than the color depth." Technical effect: Optimizes the "questioning method" for each detection, significantly improving the model's perception accuracy under complex conditions.

[0088] Innovation Point 3: Multi-stage progressive zero-shot inference process Problem: One-time "image description" has low accuracy and cannot locate.

[0089] Solution: Decompose the detection task into four sequential stages to guide the model to focus gradually: Scene analysis: The model understands the image as a whole and identifies the main components.

[0090] Anomaly detection: The model or auxiliary visual saliency algorithm identifies all regions that "look abnormal" and generates candidate region proposals.

[0091] Semantic verification: For each candidate region, after cropping, it is fed into the model, and multi-angle question-and-answer verification is performed using dynamic prompts generated by Innovation Point 2 (such as "Is this an oil stain?", "Is this a scratch?"). The text-patch matching score is calculated.

[0092] Precise localization: For high-confidence regions, issue visual localization instructions to the model (such as "Please mark all the scratches in the image with boxes") to obtain pixel-level bounding boxes.

[0093] Technical benefits: It mimics human inspection logic, improving reliability. Most importantly, it achieves defect localization under zero-sample conditions, overcoming the limitations of the native multimodal model.

[0094] Innovation Point 4: Defect Matching Algorithm Based on Cross-Modal Attention Fusion Problem: Simple global feature similarity calculation cannot capture local detailed correspondences.

[0095] Solution: In the semantic verification stage, a cross-attention mechanism is introduced. The textual description features of the defects are used as the query, and the visual feature maps of the candidate image regions are used as the key and value. Through attention calculation, the textual features can "question" different parts of the image features, thereby finding the visual evidence most relevant to the textual description.

[0096] Technical results: It achieves fine-grained region-semantic matching, which greatly improves the ability to distinguish subtle defects and complex background interference, and the matching score is more accurate.

[0097] Beneficial effects True zero-sample capability: New and old defects can be detected without collecting and labeling defect samples, greatly reducing data costs and shortening the response time to new defects from weeks to hours.

[0098] Accurate identification and localization: Through multi-stage reasoning and cross-modal attention fusion, high-precision defect identification and localization are achieved under zero-sample conditions.

[0099] Strong scene adaptability and high robustness: The dynamic prompting engine enables the system to adapt to complex field environments such as lighting, angle, and component differences, effectively reducing false alarms.

[0100] The process is interpretable and the results are reliable: It provides visualization based on semantic matching and attention focus, making the detection results easy for domain experts to understand and verify.

[0101] System self-evolution and sustainable improvement: Through confidence calibration and incremental learning mechanisms, the system performance is continuously optimized during use, forming a virtuous cycle.

[0102] This application also provides a specific application example: 1. Use trackside line scan cameras to acquire images of the high-speed train; 2. Evaluate the image quality of the acquired EMU images. If the brightness, contrast, sharpness, noise, etc. of the images do not meet the quality requirements, report them for manual processing and fault inspection. 3. Input the image that meets the quality requirements into the trained train component detection model, and output the category and corresponding coordinates of the train components in the image; 4. Construct a hierarchical, multi-granular fault semantic definition library according to the method in step one above, and predefine the fault mode descriptions corresponding to each component; 5. Through Grad-CAM image saliency analysis and autonomous discovery using a visual-language model, the results are merged via dual-path processing to screen out potential fault areas; 6. Based on the components detected in step 3, the image quality in step 2, and the suspected fault areas in step 5, generate dynamic prompts according to the method in step two above; 7. Input the prompt words generated in step 6 into the visual-language model, and infer the final result; 8. Visualize the results from step 7, push the results to the display terminal, and store the pseudo-label results in the database; 9. Pseudo-labeled data is converted into training data and the model is automatically iterated and trained for optimization. This process can be manually intervened to ensure data accuracy and model precision.

[0103] Combination Figure 3 This application also provides a train defect identification device applicable to the aforementioned electronic equipment. The train defect identification device may include a global scene analysis module, a local anomaly analysis module, an anomaly semantic verification module, and a spatial positioning processing module.

[0104] The global scene parsing module is used to perform global scene parsing on the target train image to obtain image scene description data. In this embodiment, the global scene parsing module can be used to execute... Figure 2 For details regarding step S110 shown, please refer to the previous description of step S110 for information about the global scene parsing module.

[0105] The local anomaly analysis module is used to perform local anomaly analysis on the target train image based on the image scene description data, obtaining a candidate anomaly region set, wherein each candidate anomaly region in the candidate anomaly region set refers to an area in the analyzed target train image where anomalies exist. In this embodiment of the application, the local anomaly analysis module can be used to execute... Figure 2 The relevant content regarding the local anomaly analysis module in step S120 shown can be found in the previous description of step S120.

[0106] The anomaly semantic verification module is used to perform anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set to obtain a defect identification result for each candidate anomaly region. The defect identification result reflects at least one of the following: defect type, confidence level, and defect discrimination criteria. In this embodiment, the anomaly semantic verification module can be used to execute... Figure 2 The relevant content regarding the abnormal semantic verification module in step S130 shown can be found in the previous description of step S130.

[0107] The spatial positioning processing module is used to determine a target anomaly region from the set of candidate anomaly regions based on the defect identification result of each candidate anomaly region, and to perform spatial positioning processing on the target anomaly region to obtain a spatial positioning result. The target anomaly region refers to a candidate anomaly region whose corresponding defect identification result satisfies pre-configured defect-related conditions. The spatial positioning result reflects the positioning information of the defect present in the target anomaly region. In this embodiment, the spatial positioning processing module can be used to execute... Figure 2 The relevant content regarding the spatial positioning processing module in step S140 shown can be found in the preceding description of step S140.

[0108] In this embodiment of the application, corresponding to the above-described method for identifying defects in high-speed trains applied to the electronic device, a computer-readable storage medium is also provided, which stores a computer program that executes the various steps of the method for identifying defects in high-speed trains when it is run.

[0109] The steps executed by the aforementioned computer program during runtime will not be described in detail here, but can be found in the explanation of the train defect identification method above.

[0110] In summary, the train defect identification method, apparatus, equipment, and medium provided in this application firstly perform global scene analysis on the target train image to obtain image scene description data; secondly, based on the image scene description data, perform local anomaly analysis on the target train image to obtain a set of candidate anomaly regions; then, perform anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set to obtain a defect identification result for each candidate anomaly region; finally, based on the defect identification result for each candidate anomaly region, determine the target anomaly region in the candidate anomaly region set, and perform spatial localization processing on the target anomaly region to obtain a spatial localization result. Based on the above, because global scene analysis, local anomaly analysis, anomaly semantic verification, and spatial localization processing are performed sequentially, defect identification and localization are multi-stage and progressive, achieving sufficient guidance for defect identification and localization, decomposing complex defect detection tasks, ensuring the accuracy of defect identification and localization, and thus improving the problem of relatively low reliability of defect identification in existing technologies.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0113] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0114] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for identifying defects in high-speed trains, characterized in that, include: Global scene analysis is performed on the target train image to obtain image scene description data; Based on the image scene description data, local anomaly analysis is performed on the target train image to obtain a set of candidate anomaly regions. Each candidate anomaly region in the set of candidate anomaly regions refers to an area in the analyzed target train image that is abnormal. Anomaly semantic verification is performed on each candidate anomaly region in the candidate anomaly region set to obtain a defect identification result for each candidate anomaly region, wherein the defect identification result is used to reflect at least one of the defect type, confidence level, and defect discrimination criteria; Based on the defect identification result of each candidate abnormal region, a target abnormal region is determined in the set of candidate abnormal regions, and spatial positioning processing is performed on the target abnormal region to obtain a spatial positioning result. The target abnormal region refers to a candidate abnormal region whose corresponding defect identification result satisfies the pre-configured defect-related conditions, and the spatial positioning result is used to reflect the positioning information of the defect existing in the target abnormal region.

2. The method for identifying defects in high-speed trains according to claim 1, characterized in that, The step of performing anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set to obtain the defect identification result for each candidate anomaly region includes: Context-aware analysis is performed on the target train image to obtain target context data, and based on the target context data, dynamic prompt words for defect identification are determined; Based on the dynamic prompts for defect identification, each candidate abnormal region in the candidate abnormal region set is subjected to abnormal semantic verification to obtain the defect identification result for each candidate abnormal region.

3. The method for identifying defects in high-speed trains according to claim 2, characterized in that, The steps of performing context-aware analysis on the target train image to obtain target context data, and determining dynamic prompts for defect identification based on the target context data, include: Context-aware analysis is performed on the target train image to obtain target context data; In a pre-determined fault semantic definition library, target description templates with relevant relationships to the target context data are identified. Based on the target description templates, dynamic prompts for defect identification are determined. The description templates in the fault semantic definition library include multi-layer description data, which includes at least scene-level description data, component-level description data, and defect-level description data. The scene-level description data reflects the overall context of the detection environment, which includes at least the shooting location, vehicle components, and environmental conditions. The component-level description data reflects the normal state characteristics of each key component of the train, which includes at least the component name and location information, standard appearance characteristics, allowable normal variation range, and relationships and spatial constraints between adjacent components. The defect-level description data reflects the semantic definition of each type of defect, which includes defect type and subclass, multi-angle visual feature description, occurrence pattern and typical location, distinguishing points from similar normal phenomena, and severity grading standards.

4. The method for identifying defects in high-speed trains according to claim 3, characterized in that, The step of performing context-aware analysis on the target train image to obtain target context data includes: A first context-aware analysis is performed on the target train image to obtain image content information, wherein the image content information is used to characterize component type, spatial layout and key visual attributes; A second context-aware analysis is performed on the target train image to obtain environmental factor information, wherein the environmental factor information is used to characterize lighting conditions, shooting angle and image sharpness; The target train image is subjected to third context-aware analysis to obtain interference factor information, wherein the interference factor information is used to characterize the degree of influence of stains, reflections, and obstructions on the detection. Based on the image content information, the environmental factor information, and the interference factor information, the target context data is determined.

5. The method for identifying defects in high-speed trains according to any one of claims 1-4, characterized in that, The step of performing global scene analysis on the target train image to obtain image scene description data includes: Obtain scene parsing prompt data configured for the target train image; Based on the scene analysis prompt data, global scene analysis is performed on the target train image to obtain image scene description data, wherein the image scene description data includes at least a component list, relative position and environmental features of each component of the train.

6. The method for identifying defects in high-speed trains according to any one of claims 1-4, characterized in that, The step of performing local anomaly analysis on the target train image based on the image scene description data to obtain a set of candidate anomaly regions includes: Obtain anomaly detection alert data configured for the target train image; Based on the anomaly detection prompt data and the image scene description data, local anomaly analysis is performed on the target train image to obtain a set of candidate anomaly regions.

7. The method for identifying defects in high-speed trains according to any one of claims 1-4, characterized in that, The steps of determining the target abnormal region from the set of candidate abnormal regions based on the defect identification result of each candidate abnormal region, and performing spatial localization processing on the target abnormal region to obtain the spatial localization result, include: Based on the defect identification results of each candidate abnormal region, the target abnormal region is determined from the set of candidate abnormal regions; Obtain a precise positioning instruction configured for the target abnormal area, and perform spatial positioning processing on the target abnormal area based on the precise positioning instruction to obtain a spatial positioning result. The precise positioning instruction includes generating a minimum bounding rectangle for linear defects, determining the main influence range for diffuse defects, and merging and segmenting regions for multi-point defects. The linear defects include scratches, and the diffuse defects include oil stains.

8. A defect identification device for high-speed trains, characterized in that, include: The global scene analysis module is used to perform global scene analysis on the target train image to obtain image scene description data; The local anomaly analysis module is used to perform local anomaly analysis on the target train image based on the image scene description data to obtain a set of candidate anomaly regions, wherein each candidate anomaly region in the set of candidate anomaly regions refers to the region in the analyzed target train image that is abnormal. An anomaly semantic verification module is used to perform anomaly semantic verification on each candidate anomaly region in the candidate anomaly region set to obtain a defect identification result for each candidate anomaly region, wherein the defect identification result is used to reflect at least one of the defect type, confidence level and defect discrimination criteria. The spatial positioning processing module is used to determine the target abnormal region in the set of candidate abnormal regions based on the defect identification result of each candidate abnormal region, and to perform spatial positioning processing on the target abnormal region to obtain a spatial positioning result. The target abnormal region refers to the candidate abnormal region whose corresponding defect identification result satisfies the pre-configured defect-related conditions. The spatial positioning result is used to reflect the positioning information of the defect existing in the target abnormal region.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the train defect identification method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains a computer program that, when executed, performs the train defect identification method according to any one of claims 1-7.

Images