Train fault detection method and device, electronic equipment and storage medium

Abstract

The invention provides a train fault detection method and device, electronic equipment and a storage medium, and relates to the technical field of industrial quality inspection, in particular to the technical field of vehicles. According to the specific implementation scheme, images shot by all detection stations in a train are obtained, and corresponding image reasoning results are obtained according to the shot images; if the attribute type of the to-be-detected target image is a first type, processing the image reasoning result according to a preset rule to obtain a detection result corresponding to the to-be-detected target image; and if the attribute type of the to-be-detected target image is a second type, processing the image reasoning result by using the multi-modal large model, and obtaining a fault result corresponding to the to-be-detected target image through the multi-modal large model. According to the invention, the accuracy and efficiency of fault detection can be improved, and the adaptive capacity and processing depth of complex faults are enhanced, so that a more comprehensive guarantee is provided for the safe operation of a train.

Description

Technical Field

[0001] The present disclosure relates to the field of industrial quality inspection technology, specifically to the field of vehicle technology, and in particular to a train fault detection method, device, electronic equipment and storage medium. Background Art

[0002] In the industrial field, defect detection is a crucial part of the entire production process. Improving product quality to produce high value-added and high-profit products can achieve a leap in product competitiveness. Ideas for improving product quality include strengthening quality inspection, improving process technology and standardizing production operations. Among them, strengthening quality inspection is the most commonly used method in manufacturing production.

[0003] Taking the railway train fault detection scenario as an example, there are many types of faults that need to be detected in the railway train fault detection scenario, including a large number of complex faults that require common sense to identify. However, the amount of such fault data is small and usually cannot be identified by local features. Therefore, the problem of how to successfully identify various faults needs to be solved urgently. Summary of the invention

[0004] The present invention provides a train fault detection method, device, electronic equipment and storage medium.

[0005] According to one aspect of the present disclosure, a train fault detection method is provided, comprising:

[0006] Acquire images taken by each inspection station in the train, and obtain corresponding image inference results based on the taken images, wherein the image inference results include prediction results corresponding to the taken images;

[0007] If the attribute type of the target image to be detected is the first type, the image inference result is processed according to a preset rule to obtain a detection result corresponding to the target image to be detected;

[0008] If the attribute type of the target image to be detected is the second type, the multimodal large model is simultaneously used to process the image reasoning result, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model.

[0009] According to another aspect of the present disclosure, there is provided a train fault detection device, comprising:

[0010] An acquisition module is used to acquire images taken by each inspection station in the train, and obtain corresponding image reasoning results based on the taken images, wherein the image reasoning results include prediction results corresponding to the taken images;

[0011] A first processing module, configured to process the image inference result according to a preset rule to obtain a detection result corresponding to the target image to be detected if the attribute type of the target image to be detected is the first type;

[0012] The second processing module is used to simultaneously use a multimodal large model to process the image reasoning result if the attribute type of the target image to be detected is the second type, and obtain a fault result corresponding to the target image to be detected through the multimodal large model.

[0013] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any method in any of the above technical solutions.

[0017] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any one of the methods described in the above technical solutions.

[0018] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the methods described in the above technical solutions when executed by a processor.

[0019] The present disclosure provides a train fault detection method, device, equipment and storage medium. When the present disclosure performs fault detection on a target image to be detected whose attribute type is the first type, the image inference result is processed according to a preset rule. For a target image to be detected of the second type, a multimodal large model is used to analyze the target image to be detected after adopting the same processing method as the first type, and a detection result corresponding to the target image to be detected is obtained through the multimodal large model. For target faults to be detected of the second type of attribute type, the multimodal large model is used to assist in in-depth analysis, thereby ensuring that complex faults can be accurately identified. It can be seen that this method can not only improve the accuracy and efficiency of fault detection, but also enhance the adaptability and processing depth to complex faults, thereby providing more comprehensive protection for the safe operation of trains.

[0020] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0022] Figure 1 is a schematic diagram of the steps of the train fault detection method in an embodiment of the present disclosure;

[0023] Figure 2 is a schematic diagram of a fault detection process for a target image to be detected whose attribute type is the first type in one embodiment of the present disclosure;

[0024] Figure 3 is a schematic diagram of a fault detection process for a target image to be detected whose attribute type is the second type in one embodiment of the present disclosure;

[0025] Figure 4 is a schematic diagram of a process of performing fault detection on a target image to be detected in one embodiment of the present disclosure;

[0026] Figure 5 A principle block diagram of a train fault detection device in an embodiment of the present disclosure;

[0027] Figure 6 It is a block diagram of an electronic device used to implement the train fault detection method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0029] Existing train fault detection methods can be mainly divided into the following types: the first method is to use manual fault detection, that is, to manually inspect each component that needs to be inspected; the second method is to perform fault detection based on traditional image processing. The specific method is: use traditional image processing algorithms to process the pictures collected by the camera, quantify the defects, and detect the defects that meet the conditions by setting thresholds; the third method is to perform fault detection based on general deep learning technology.

[0030] However, the above methods have the following shortcomings: For the first method, manual defect detection requires workers to stay at a fixed workstation for a long time and observe the product by naked eyes to determine whether there are defects. In this way, not only is the labor intensity extremely high, but the detection stability and consistency are also poor. In addition, the manual quality inspection method generally uses paper and pen to record the quality inspection results, which results in incomplete and scattered data obtained from the inspection, and cannot form valuable feedback information to guide lean production. For the second method, the defect detection method based on traditional image processing generally requires feature design for specific products, and a series of rules are defined according to shape, color, width, height, etc. to perform defect detection. The robustness of this method is poor. With the change of the object to be detected, all rules and algorithms must be redesigned and developed, and even different batches of the same product cannot reuse a set of rules. For the third method, the existing defect detection methods based on deep learning technology generally directly reuse the object detection and segmentation algorithms of general images. There is no specific algorithm designed for the specific scenario of defect detection using existing prior knowledge, and there is still a lot of room for improvement in effect. In addition, the existing deep learning solutions are basically single-model design post-processing and other processing processes, and do not have the ability to make comprehensive decisions based on multiple models.

[0031] In order to solve the above technical problems, the present invention provides a new train fault detection method. Figure 1 As shown, the method includes:

[0032] Step S101, obtaining images taken by each inspection station in the train, and obtaining corresponding image inference results based on the taken images, wherein the image inference results include prediction results corresponding to the taken images.

[0033] Specifically, when detecting a train fault, multiple inspection stations are generally used to shoot the bottom and side of the vehicle body, so that images taken at different stations can be obtained. After obtaining the images taken at different stations, the images taken at different stations are generally input into different target detection models for inference and prediction to obtain image inference results corresponding to the images taken at different stations. Among them, the image inference results include prediction results corresponding to the captured images, which may include the category of objects contained in the captured images, and of course, other key information contained in the captured images. In this way, by obtaining the image inference results corresponding to the images taken at each inspection station in advance, it is convenient for subsequent fault judgment.

[0034] Step S102: If the attribute type of the target image to be detected is the first type, the image inference result is processed according to a preset rule to obtain a detection result corresponding to the target image to be detected.

[0035] Specifically, the target image to be detected is an image to be judged, and whether there is a fault to be judged in the image, wherein there are multiple target images to be detected. After obtaining the image reasoning result, different fault detection methods can be determined according to the type of the target image to be detected. Therefore, before determining different fault detection methods, the attribute type of the target image to be detected can be judged. If the attribute type of the target image to be detected is the first type, a preset rule is used to perform a simple and controllable explicit logical judgment on the image reasoning result, so that the detection result corresponding to the target image to be detected can be, for example, the presence of a fault, or the absence of a fault.

[0036] The preset rules may be pre-set post-processing operations, and the preset rules may include specific rules such as name matching, threshold filtering, etc. Referring to Table 1 below, the specific preset rules may include the contents in Table 1.

[0037]

[0038] Table 1

[0039] In this way, the image reasoning results are processed by the preset rules, so that the detection results corresponding to the target image to be detected can be obtained. In addition, the correspondence between the preset rules and the image reasoning results, that is, which image reasoning result is processed by which preset rule is pre-set, and those skilled in the art can set it according to actual needs, which is not limited here.

[0040] Step S103: If the attribute type of the target image to be detected is the second type, the multimodal large model is simultaneously used to process the image reasoning result, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model.

[0041] Specifically, it should be noted that the first type and the second type have different attributes. Specifically, the first type can be considered as a simple type, while the second type is a complex type. The division of simple types and complex types can be specifically divided according to actual applications. For example, the first type and the second type are divided in advance. As long as a certain component structure in the train is involved, it is considered to be a complex type, that is, the second type, while other component structures are considered to be simple types, that is, the first type. Therefore, the attribute type of the target image to be detected can be obtained by judging whether there is a specified component structure object in the image to be detected.

[0042] In the embodiment of the present disclosure, if the attribute type of the target image to be detected is the second type, it is necessary to process the image inference result with the help of a multimodal large model at the same time, that is, first use the same processing method as the first type, and then use the multimodal large model to obtain the fault result corresponding to the target image to be detected. The detection result corresponding to the target image to be detected may be, for example, the presence of a fault, or the absence of a fault.

[0043] The present disclosure provides a train fault detection method, device, equipment and storage medium. When the present disclosure performs fault detection on a target image to be detected whose attribute type is the first type, the image inference result is processed according to a preset rule. For a target image to be detected of the second type, a multimodal large model is used to analyze the target image to be detected after adopting the same processing method as the first type, and a detection result corresponding to the target image to be detected is obtained through the multimodal large model. For target faults to be detected of the second type of attribute type, the multimodal large model is used to assist in in-depth analysis, thereby ensuring that complex faults can be accurately identified. It can be seen that this method can not only improve the accuracy and efficiency of fault detection, but also enhance the adaptability and processing depth to complex faults, thereby providing more comprehensive protection for the safe operation of trains.

[0044] In some optional embodiments, images captured by each inspection station in the train are obtained, and corresponding image reasoning results are obtained according to the captured images, including:

[0045] Obtain images taken by each inspection station in the train;

[0046] The images taken at each inspection station are respectively input into different target detection models, and the image inference results corresponding to the images taken at each inspection station are obtained through the target detection model.

[0047] Specifically, when detecting a train fault, multiple inspection stations are generally used to photograph the bottom and side of the vehicle to obtain images taken at different stations. After obtaining the images taken at different stations, the images taken at different stations are generally input into different models for inference and prediction to obtain image inference results corresponding to the images at different stations. The image inference results include prediction results corresponding to the captured images, which may include the category of the object contained in the captured images, and of course, other key information contained in the captured images.

[0048] In this way, by obtaining the image inference results corresponding to the images taken by each inspection station in advance, it is convenient to make a subsequent fault judgment.

[0049] In some optional embodiments, if the attribute type of the target image to be detected is the first type, the image inference result is processed according to a preset rule to obtain a detection result corresponding to the target image to be detected, including:

[0050] If the attribute type of the target image to be detected is the first type, parsing the business logic contained in the target image to be detected to obtain a business parsing result;

[0051] Determine the image reasoning result to be used in the image reasoning result according to the business analysis result, and obtain the first target image reasoning result;

[0052] The first target image inference result is processed according to preset rules to obtain a detection result corresponding to the target image to be detected.

[0053] Specifically, if the attribute type of the target image to be detected is the first type, the business logic contained in the target image to be detected is parsed to obtain the business parsing result, and then the first target image reasoning result is determined according to the business parsing result. The business logic contained in the target image to be detected is used to indicate which image reasoning results corresponding to the workstation images are needed to jointly judge whether the target image to be detected exists in order to determine whether the fault exists. For example, taking the detection of whether the anti-theft device is damaged in the target image to be detected as an example, by parsing the target image to be detected, it can be obtained that the business logic contained in the target image to be detected is that the anti-theft device is tied with wire, then the anti-theft device is damaged. At this time, it can be known that the image reasoning results obtained by inputting the target image to be detected into the wire detection model and the position positioning detection model respectively are needed, and then these two image reasoning results are used as the first target image reasoning result. After obtaining the first target image reasoning result, the first target image reasoning result is processed according to the preset rules, and finally the detection result corresponding to the target image to be detected can be obtained.

[0054] In this way, by analyzing the business logic contained in the target image to be detected, it is helpful to determine which image reasoning results need to be processed, that is, it is helpful to determine the first target image reasoning result, which in turn helps to improve fault detection efficiency.

[0055] In some optional embodiments, the first target image inference result is processed according to a preset rule to obtain a detection result corresponding to the target image to be detected, including:

[0056] If the first target image reasoning result includes at least two image reasoning results, each image reasoning result in the first target image reasoning result is processed respectively using a preset rule to obtain a processed image reasoning result;

[0057] Perform logical operations on the processed image reasoning results to obtain the detection results corresponding to the target image to be detected.

[0058] Specifically, after obtaining the target image reasoning result, it is necessary to determine which image reasoning results are included in the first target image reasoning result. If the first target image reasoning result includes at least two image reasoning results, each image reasoning result in the target image reasoning result is processed separately using a preset rule to obtain a processed image reasoning result, and then a logical operation is performed on the processed image reasoning result. See Table 2, which shows that logical operations include logical AND, logical OR, and logical NOT.

[0059] Table 2

[0060] After obtaining the processed image processing results, the processed image processing results are subjected to logical operations, thereby facilitating obtaining the detection result corresponding to the target image to be detected. If the target image reasoning result only includes one image reasoning result, the detection result corresponding to the target image to be detected can be directly obtained based on the image reasoning result.

[0061] In this way, according to the number of image reasoning results included in the target image reasoning results, different logical operations are performed on the image reasoning results, which is helpful to accurately determine whether a fault exists and improve the accuracy of fault judgment.

[0062] For the convenience of the present application, the following examples are given: Figure 2 , Figure 2 It is a schematic diagram of a fault detection process for a target image to be detected whose attribute type is the first type in an embodiment of the present disclosure. Taking whether there is a fault of anti-theft device damage in the target image to be detected as an example, by parsing the target image to be detected, it can be obtained that the business logic contained in the target image to be detected is that the anti-theft device is tied with wire, and the anti-theft device is damaged. At this time, it can be known that the image reasoning results obtained by the wire detection model and the part positioning detection model are needed, and the wire model reasoning results and the component positioning reasoning results are first obtained. After obtaining the wire model reasoning results and the component positioning reasoning results, these two image reasoning results are used as the target image reasoning results. Since the target image reasoning result includes two image reasoning results, the two image reasoning results are respectively subjected to name matching, threshold filtering, NMS (non-maximum suppression), and IOU (intersection-over-union) calculation operations, and the image reasoning results after each processing are obtained, which are wire and anti-theft device. Next, the two processed image reasoning results are logically processed. Since the wire and the anti-theft device exist at the same time, it can be obtained that the target image to be detected exists, that is, the anti-theft device is damaged.

[0063] In some optional embodiments, if the attribute type of the target image to be detected is the second type, the multimodal large model is simultaneously used to process the image reasoning result, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model, including:

[0064] If the attribute type of the target image to be detected is the second type, then after adopting the same processing method as the first type, the second target image inference result is determined and the prompt word is generated according to the business logic contained in the target image to be detected;

[0065] The prompt word and the second target image inference result are input into the multimodal large model, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model.

[0066] Specifically, if the attribute type of the target image to be detected is the second type, then after adopting the same processing method as the first type, it is necessary to use the multimodal large model to process the image reasoning result. Specifically, the second target image reasoning result is first determined according to the business logic contained in the target image to be detected and a prompt word is generated. Then the prompt word and the second target image reasoning result are input into the multimodal large model, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model. Among them, it should be noted that the multimodal large language model is a new research hotspot that has emerged in recent years. It uses a powerful large language model as the brain to perform multimodal tasks. Since the multimodal large model adopted in the embodiment of the present disclosure is similar to the large model training method in the prior art, the training method of the multimodal large model in the present disclosure will not be repeated here.

[0067] For example, in the train fault detection system, there is a fault type called "interlocking difference out of limit", which refers specifically to the abnormality of the interlocking difference component, and usually only the interlocking difference component will have this type of fault. If a new target image to be detected is received, it will first be processed according to the simple fault type, that is, it will be processed in the same way as the first type, because many faults can be quickly judged through direct feature recognition. However, if the component detection model identifies the interlocking difference component in the image, the system will take an additional step, that is, it will be analyzed in depth again through the multimodal large model branch of complex faults. This branch uses advanced multimodal data processing technology, combined with image content and prior knowledge of components, to comprehensively judge whether the interlocking difference component has an out-of-limit fault condition. This double-check mechanism ensures that even complex and difficult to intuitively identify faults can be accurately detected and identified, thereby improving the reliability and safety of train fault detection.

[0068] In this way, for the target faults to be detected with the second attribute type, an efficient multi-modal large model is used for auxiliary judgment, which realizes complex fault discrimination without consuming too much computing resources, thereby improving the efficiency of fault detection. In this way, not only the accuracy and efficiency of fault detection can be improved, but also the adaptability and processing depth of complex faults can be enhanced, thereby providing more comprehensive protection for the safe operation of trains.

[0069] In some optional embodiments, determining the second target image inference result and generating the prompt word according to the business logic contained in the target image to be detected includes:

[0070] Analyze the business logic contained in the target image to be detected;

[0071] Determine the image reasoning result to be used according to the business logic obtained by the analysis, and obtain the second target image reasoning result;

[0072] The prompt word is generated using a prompt word generation template according to the business logic obtained by the analysis.

[0073] Specifically, when generating prompt words, a prompt word generation template is first obtained according to a priori fault judgment criteria, and then a prompt word (also referred to as Prompt) is generated using the prompt word generation template.

[0074] In this way, by analyzing the business logic obtained to determine the inference result of the second target image and the prompt word, complex fault identification can be achieved without consuming too much computing power resources, which is conducive to the judgment of complex faults and further improves the efficiency of fault detection.

[0075] For the convenience of the present application, the following examples are given: Figure 3 , Figure 3 This is a schematic diagram of a fault detection process for a target image to be detected whose attribute type is the second type in one embodiment of the present disclosure. Assuming that it is determined whether there is a cross-hook difference over-limit fault in the target image to be detected, after parsing the target image to be detected, it is obtained that the business logic contained in the target image to be detected is that the cross-hook difference fault occurs when the front and rear carriage coupler connection positions of the cross-hook difference station deviate up and down, then the following can be done: Figure 3The implicit logic judgment process of the multimodal large model is shown. First, through the business logic obtained by parsing, it can be known that the image reasoning result corresponding to the component positioning detection model is needed, and the image reasoning result is used as the second target image reasoning result. Then, the prompt word corresponding to the fault is generated according to the business judgment logic. The prompt word can be obtained through the above business logic: If the position of the hook difference between the front and rear carriages is offset up and down, there is a "hooking difference over-limit" fault. Does the hook difference in the figure have this fault? Then, the prompt word and the second target image reasoning result are input into the multimodal large model. The fault judgment result obtained by the multimodal large model, that is, the answer output by the multimodal large model is: "Yes, there is a hook difference over-limit fault."

[0076] In some optional embodiments, before acquiring images captured by each inspection station in the train and obtaining corresponding image reasoning results based on the captured images, the method further includes:

[0077] Acquire the target image to be detected;

[0078] If the target image to be detected does not contain the specified component structure object, the attribute type of the target image to be detected is the first type;

[0079] If the target image to be detected contains the specified component structure object, the attribute type of the target image to be detected is the second type.

[0080] Specifically, in the train inspection process, the disclosed embodiment first obtains images taken by each inspection station, and then analyzes these images to determine the attribute type of the target image to be inspected. Specifically, first check whether the target image to be inspected contains a specific component structure image: if the specified component structure is not included, then the attribute type of this image is considered to be the first type, that is, a simple type fault, which usually has obvious fault characteristics and is easy to identify through preset rules; if the image contains a specified component structure, then it is classified as the second type, that is, a complex type fault, which may require more advanced analysis techniques to identify. Through this classification method, the system can apply appropriate detection strategies for different types of fault images, thereby improving the accuracy and efficiency of fault detection.

[0081] For example, in the train fault detection system, there is a fault type called "interlocking difference exceeding the limit", which refers specifically to the abnormality of the interlocking difference component, and usually only the interlocking difference component will have this type of fault. If a new target image to be detected contains an interlocking difference component, the attribute type of the target image to be detected is classified as the second type. For this type of image, it will first be processed according to the simple fault type, and then in-depth analysis will be performed through the multi-modal large model branch of complex faults.

[0082] In this way, images taken by various inspection stations in the train are obtained, and the types of target images to be inspected are divided before the corresponding image inference results are obtained based on the taken images, which is conducive to quickly determining the fault judgment method to be adopted.

[0083] In order to understand the technical solution of the present disclosure as a whole, see Figure 4 , Figure 4 The figure is a flow chart of fault detection for a target image to be detected in one embodiment of the present disclosure. In this method, images taken by each detection station in the train are first obtained, and corresponding image inference results are obtained based on the captured images. The image inference results include prediction results corresponding to the captured images. The image inference results include Figure 2 The 1-station image reasoning results, 2-station image reasoning results, and N-station image reasoning results.

[0084] If the attribute type of the target image to be detected is the first type, the business logic contained in the target image to be detected is parsed to obtain the business parsing result; the image reasoning result to be used in the image reasoning result is determined according to the business parsing result to obtain the first target image reasoning result. The image reasoning results contained in the first target image reasoning result are respectively subjected to name matching, threshold filtering, NMS, and IOU (intersection over union) calculation operations to obtain the image reasoning results after each processing. Then, according to actual needs, the image reasoning results after each processing are subjected to logical operation processing, wherein the logical operation includes any one of logical AND, logical OR, and logical NOT, so as to obtain the detection result corresponding to the target image to be detected.

[0085] If the type of the target image to be detected is the second type, then after processing in the first type of processing method, the target image to be detected is parsed to obtain the business logic contained in the target image to be detected. The business logic obtained by parsing can know the image reasoning result that needs to be used, and the image reasoning result is used as the second target image reasoning result. Then, according to the business judgment logic, the prompt word corresponding to the fault is generated. The prompt word can be obtained through the above business logic, for example: whether there is a fault in component X in the image according to a certain fault standard. Then, the prompt word and the second target image reasoning result are input into the multimodal large model, and the fault judgment result can be obtained through the multimodal large model.

[0086] The disclosed embodiments start from the characteristics of railway scene faults and application scenarios, and aim at the fact that fault types in real train scenarios are complex and have business dependencies with each other. A multi-modal large model joint prediction method for train fault detection is proposed, which realizes the detection capability of cross-graph, cross-camera faults and the implicit logic judgment capability of complex faults.

[0087] The following describes an apparatus embodiment of the present application, which can be used to execute the train fault detection method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the train fault detection method of the present application.

[0088] The present disclosure also provides a train fault detection device 500, such as Figure 5 As shown, including:

[0089] The acquisition module 501 is used to acquire images taken by each inspection station in the train, and obtain corresponding image reasoning results according to the taken images, wherein the image reasoning results include prediction results corresponding to the taken images;

[0090] A first processing module 502 is used to process the image inference result according to a preset rule to obtain a detection result corresponding to the target image to be detected if the attribute type of the target image to be detected is the first type;

[0091] The second processing module 503 is used to process the image reasoning result by using the multimodal large model at the same time if the attribute type of the target image to be detected is the second type, and obtain the fault result corresponding to the target image to be detected through the multimodal large model.

[0092] In some optional embodiments, the acquisition module 501 acquires images captured by each inspection station in the train, and obtains corresponding image reasoning results according to the captured images, including:

[0093] Obtain images taken by each inspection station in the train;

[0094] The images taken at each inspection station are respectively input into different target detection models, and the image inference results corresponding to the images taken at each inspection station are obtained through the target detection model.

[0095] In some optional embodiments, if the attribute type of the target image to be detected is the first type, the first processing module 502 processes the image reasoning result according to a preset rule to obtain a detection result corresponding to the target image to be detected, including:

[0096] If the type of the target image to be detected is the first type, parsing the business logic contained in the target image to be detected to obtain a business parsing result;

[0097] Determine the image reasoning result to be used in the image reasoning result according to the business analysis result, and obtain the first target image reasoning result;

[0098] The first target image inference result is processed according to preset rules to obtain a detection result corresponding to the target image to be detected.

[0099] In some optional embodiments, the first processing module 502 processes the first target image inference result according to a preset rule to obtain a detection result corresponding to the target image to be detected, including:

[0100] If the first target image reasoning result includes at least two image reasoning results, each image reasoning result in the first target image reasoning result is processed respectively using a preset rule to obtain a processed image reasoning result;

[0101] Perform logical operations on the processed image reasoning results to obtain the detection results corresponding to the target image to be detected.

[0102] In some optional embodiments, if the attribute type of the target image to be detected is the second type, the second processing module 503 simultaneously uses the multimodal large model to process the image reasoning result, and obtains the fault result corresponding to the target image to be detected through the multimodal large model, including:

[0103] If the attribute type of the target image to be detected is the second type, then after adopting the same processing method as the first type, the second target image inference result is determined and the prompt word is generated according to the business logic contained in the target image to be detected;

[0104] The prompt word and the second target image inference result are input into the multimodal large model, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model.

[0105] In some optional embodiments, the second processing module 503 determines the second target image reasoning result and generates a prompt word according to the business logic contained in the target image to be detected, including:

[0106] Analyze the business logic contained in the target image to be detected;

[0107] Determine the image reasoning result to be used according to the business logic obtained by the analysis, and obtain the second target image reasoning result;

[0108] The prompt word is generated using a prompt word generation template according to the business logic obtained by the analysis.

[0109] In some optional embodiments, the acquisition module 501 is further used to acquire the image of the target to be detected before acquiring the images captured by each detection station in the train and obtaining the corresponding image reasoning results according to the captured images;

[0110] If the target image to be detected does not contain the specified component structure object, the attribute type of the target image to be detected is the first type;

[0111] If the target image to be detected contains the specified component structure object, the attribute type of the target image to be detected is the second type.

[0112] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0113] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0114] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0115] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0116] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0117] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the train fault detection method. For example, in some embodiments, the train fault detection method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the applet distribution described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the train fault detection method in any other appropriate manner (e.g., by means of firmware).

[0118] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0119] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0120] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0123] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0124] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0125] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A train fault detection method, comprising: Acquire images taken by each inspection station in the train, and obtain corresponding image inference results based on the taken images, wherein the image inference results include prediction results corresponding to the taken images; If the attribute type of the target image to be detected is the first type, the image inference result is processed according to a preset rule to obtain a detection result corresponding to the target image to be detected; If the attribute type of the target image to be detected is the second type, the multimodal large model is simultaneously used to process the image reasoning result, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model.

2. The method according to claim 1, wherein: The acquiring of images captured by each inspection station in the train and obtaining corresponding image reasoning results according to the captured images include: Obtain images taken by each inspection station in the train; The images captured by the various detection stations are respectively input into different target detection models, and image inference results corresponding to the images captured by the various detection stations are obtained through the target detection models.

3. The method according to claim 1, wherein: If the attribute type of the target image to be detected is the first type, the image inference result is processed according to a preset rule to obtain a detection result corresponding to the target image to be detected, including: If the type of the target image to be detected is the first type, parsing the business logic contained in the target image to be detected to obtain a business parsing result; Determine the image reasoning result to be used in the image reasoning result according to the business analysis result, and obtain a first target image reasoning result; The first target image inference result is processed according to the preset rule to obtain a detection result corresponding to the target image to be detected.

4. The method according to claim 3, wherein: The step of processing the first target image inference result according to the preset rule to obtain a detection result corresponding to the target image to be detected includes: If the first target image reasoning result includes at least two image reasoning results, each image reasoning result in the first target image reasoning result is processed respectively by using the preset rule to obtain a processed image reasoning result; Performing logical operations on the processed image reasoning results to obtain detection results corresponding to the target image to be detected.

5. The method according to claim 1, wherein: If the attribute type of the target image to be detected is the second type, the image reasoning result is processed by using a multimodal large model at the same time, and a fault result corresponding to the target image to be detected is obtained by using the multimodal large model, including: If the attribute type of the target image to be detected is the second type, after adopting the same processing method as the first type, the second target image inference result is determined and the prompt word is generated according to the business logic contained in the target image to be detected; The prompt word and the second target image inference result are input into the multimodal large model, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model.

6. The method according to claim 5, wherein: The determining the second target image inference result and generating the prompt word according to the business logic contained in the target image to be detected includes: Analyze the business logic contained in the target image to be detected; Determine the image reasoning result to be used according to the business logic obtained by the analysis, and obtain the second target image reasoning result; The prompt word is generated using a prompt word generation template according to the business logic obtained by the analysis.

7. The method according to any one of claims 1 to 6, wherein: Before acquiring images captured by each inspection station in the train and obtaining corresponding image inference results based on the captured images, the method further includes: Acquire the target image to be detected; If the target image to be detected does not contain the specified component structure object, the attribute type of the target image to be detected is the first type; If the target image to be detected contains a designated component structure object, the attribute type of the target image to be detected is the second type.

8. A train fault detection device, comprising: An acquisition module is used to acquire images taken by each inspection station in the train, and obtain corresponding image reasoning results based on the taken images, wherein the image reasoning results include prediction results corresponding to the taken images; A first processing module, configured to process the image inference result according to a preset rule to obtain a detection result corresponding to the target image to be detected if the attribute type of the target image to be detected is the first type; The second processing module is used to simultaneously use a multimodal large model to process the image reasoning result if the attribute type of the target image to be detected is the second type, and obtain a fault result corresponding to the target image to be detected through the multimodal large model.

9. The device according to claim 8, wherein: The acquisition module acquires images captured by each detection station in the train and obtains corresponding image reasoning results based on the captured images, including: Obtain images taken by each inspection station in the train; The images captured by the various detection stations are respectively input into different target detection models, and image inference results corresponding to the images captured by the various detection stations are obtained through the target detection models.

10. The device according to claim 8, wherein: If the attribute type of the target image to be detected is the first type, the first processing module processes the image reasoning result according to a preset rule to obtain a detection result corresponding to the target image to be detected, including: If the type of the target image to be detected is the first type, parsing the business logic contained in the target image to be detected to obtain a business parsing result; Determine the image reasoning result to be used in the image reasoning result according to the business analysis result, and obtain a first target image reasoning result; The first target image inference result is processed according to the preset rule to obtain a detection result corresponding to the target image to be detected.

11. The device according to claim 10, wherein: The first processing module processes the first target image inference result according to the preset rule to obtain a detection result corresponding to the target image to be detected, including: If the first target image reasoning result includes at least two image reasoning results, each image reasoning result in the first target image reasoning result is processed respectively by using the preset rule to obtain a processed image reasoning result; Performing logical operations on the processed image reasoning results to obtain detection results corresponding to the target image to be detected.

12. The device according to claim 8, wherein: If the attribute type of the target image to be detected is the second type, the second processing module simultaneously uses the multimodal large model to process the image reasoning result, and obtains the fault result corresponding to the target image to be detected through the multimodal large model, including: If the attribute type of the target image to be detected is the second type, then after adopting the same processing method as the first type, the second target image inference result is determined and the prompt word is generated according to the business logic contained in the target image to be detected; The prompt word and the second target image inference result are input into the multimodal large model, and the fault result corresponding to the target image to be detected is obtained through the multimodal large model.

13. The device according to claim 12, wherein: The second processing module determines the second target image reasoning result and generates a prompt word according to the business logic contained in the target image to be detected, including: Analyze the business logic contained in the target image to be detected; Determine the image reasoning result to be used according to the business logic obtained by the analysis, and obtain the second target image reasoning result; The prompt word is generated using a prompt word generation template according to the business logic obtained by the analysis.

14. The device according to any one of claims 8 to 13, wherein: The acquisition module is further used to acquire the target image to be detected before acquiring the images captured by each detection station in the train and obtaining the corresponding image reasoning results based on the captured images; If the target image to be detected does not contain the specified component structure object, the attribute type of the target image to be detected is the first type; If the target image to be detected contains a designated component structure object, the attribute type of the target image to be detected is the second type.

15. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

17. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images